docs: add SQL-only schema update script for v2610.1

update(2610.1).sql brings an existing database up to the current schema without PHP: base tables (CREATE IF NOT EXISTS), the v2603.x migrations (last_event_at column, uq_event removal) and the v2610.1 index changes (idx_occurred, idx_src_ip_occ, drop idx_src_ip). All steps are idempotent via information_schema checks + PREPARE/EXECUTE, so the file is safe to re-run and portable across MySQL 8 and MariaDB.
release: bump version to 2610.1
2026-10-19 12:12:00 -04:00 · 2026-10-18 23:43:03 -04:00 · 2026-10-18 22:18:08 -04:00 · 2026-10-18 22:12:18 -04:00 · 2026-10-18 22:01:03 -04:00 · 2026-10-18 21:56:33 -04:00
6 changed files with 285 additions and 36 deletions
@@ -42,6 +42,7 @@ Each portspoof_py instance runs independently and exposes a JSON API. portspoof_
 portspoof_concentrator/
 ├── config.php            Database credentials and app-wide constants
 ├── schema.sql            MySQL DDL (applied by setup.php)
+├── update(2610.1).sql    SQL-only schema update for existing databases (this release)
 ├── setup.php             One-time install / migration script
 ├── index.php             Aggregated dashboard
 ├── nodes.php             Add / edit / delete portspoof_py nodes
@@ -225,7 +226,7 @@ One row per connection event ingested from any node.
 | `banner_len` | `INT UNSIGNED` | Length of the banner in bytes |
 | `created_at` | `DATETIME` | Row insertion time |

-**Indexes:** `(node_id, occurred_at)` for cursor lookups and per-node queries; `src_ip` and `dst_port` for the top-N aggregations.
+**Indexes:** `(node_id, occurred_at)` for cursor lookups and per-node queries; `occurred_at` for time-window queries, the recent-connections sort, and the purge; `(src_ip, occurred_at)` for the frequent-IP aggregation; `dst_port` for the top-port aggregation.

 ---

@@ -501,13 +502,21 @@ curl -H "Authorization: Bearer your-token" "https://yourserver/api/connections.p

 ## Upgrading

-If you deployed before `last_event_at` was added to the schema, run `setup.php` again:
+When upgrading, migrate the database to the new schema with either of:
+
+**Via the PHP setup script:**

 ```bash
 php /var/www/portspoof_concentrator/setup.php
 ```

-It will add the missing column and drop the legacy `uq_event` unique key if present. No data is lost.
+**Via SQL (no PHP required):**
+
+```bash
+mysql -u portspoof -p portspoof_concentrator < "update(2610.1).sql"
+```
+
+Both apply all pending migrations idempotently (the `last_event_at` column, the legacy `uq_event` key removal, and the `occurred_at` / `(src_ip, occurred_at)` indexes) and can be re-run safely. No data is lost. Note that on a large `connections` table the `ADD KEY` steps take time proportional to the table size.

 ---

@@ -75,19 +75,16 @@ function fetch_node_api(array $node, string $path): mixed {
 * Ingest connection events from portspoof_py that are strictly newer than
 * $last_event_at (ISO 8601 string, or null to ingest everything).
 *
+ * New rows are written with multi-row INSERTs in chunks, inside a single
+ * transaction per call, to keep the number of round trips and commits low.
+ *
 * Returns ['inserted' => int, 'new_max_ts' => string|null]
 * where new_max_ts is the DATETIME(6) of the newest row inserted, or null if
 * nothing new was inserted.
 */
 function ingest_connections(int $node_id, array $events, ?string $last_event_at): array {
-    $inserted   = 0;
    $new_max_ts = null;
-
-    $s = db()->prepare(
-        'INSERT INTO connections
-         (node_id, occurred_at, src_ip, src_port, dst_port, banner_hex, banner_len)
-         VALUES (?, ?, ?, ?, ?, ?, ?)'
-    );
+    $rows       = [];

    foreach ($events as $ev) {
        $raw_ts = $ev['timestamp'] ?? null;
@@ -103,7 +100,7 @@ function ingest_connections(int $node_id, array $events, ?string $last_event_at)
            continue;
        }

-        $s->execute([
+        $rows[] = [
            $node_id,
            $ts,
            $ev['src_ip']   ?? '',
@@ -111,15 +108,40 @@ function ingest_connections(int $node_id, array $events, ?string $last_event_at)
            (int)($ev['dst_port']  ?? 0),
            $ev['banner_hex'] ?? null,
            (int)($ev['banner_len'] ?? 0),
-        ]);
-        $inserted++;
+        ];

        if ($new_max_ts === null || $ts > $new_max_ts) {
            $new_max_ts = $ts;
        }
    }

-    return ['inserted' => $inserted, 'new_max_ts' => $new_max_ts];
+    if ($rows === []) {
+        return ['inserted' => 0, 'new_max_ts' => null];
+    }
+
+    $pdo = db();
+    $pdo->beginTransaction();
+    try {
+        foreach (array_chunk($rows, 200) as $chunk) {
+            $values = implode(', ', array_fill(0, count($chunk), '(?, ?, ?, ?, ?, ?, ?)'));
+            $params = [];
+            foreach ($chunk as $r) {
+                $params = array_merge($params, $r);
+            }
+            $s = $pdo->prepare(
+                'INSERT INTO connections
+                 (node_id, occurred_at, src_ip, src_port, dst_port, banner_hex, banner_len)
+                 VALUES ' . $values
+            );
+            $s->execute($params);
+        }
+        $pdo->commit();
+    } catch (PDOException $e) {
+        $pdo->rollBack();
+        throw $e;
+    }
+
+    return ['inserted' => count($rows), 'new_max_ts' => $new_max_ts];
 }

 /**
@@ -236,15 +258,34 @@ function frequent_ips(int $min_connections, ?string $since = null): array {

 /**
 * Delete connections older than retention_days setting.
+ * Rows are removed in batches so each transaction stays small — a single
+ * giant DELETE would lock a large part of the table while the fetch cron
+ * keeps inserting rows.
+ *
 * Returns the number of rows deleted.
 */
 function purge_old_connections(): int {
    $days   = max(1, (int)get_setting('retention_days', '7'));
+    $cutoff = date('Y-m-d H:i:s', time() - $days * 86400);
+
    $s = db()->prepare(
-        'DELETE FROM connections WHERE occurred_at < NOW() - INTERVAL ? DAY'
+        'DELETE FROM connections
+         WHERE occurred_at < ?
+         ORDER BY id
+         LIMIT 5000'
    );
-    $s->execute([$days]);
-    return $s->rowCount();
+
+    $total = 0;
+    do {
+        $s->execute([$cutoff]);
+        $deleted = $s->rowCount();
+        $total += $deleted;
+        if ($deleted > 0) {
+            usleep(100_000); // 100 ms between batches to ease lock pressure
+        }
+    } while ($deleted >= 5000);
+
+    return $total;
 }

 // ── Upstream version check ────────────────────────────────────────────────────
@@ -252,18 +293,24 @@ function purge_old_connections(): int {
 define('UPSTREAM_VERSION_URL', 'https://git.ny.daprogs.com/api/v1/repos/DAProgs/portspoof_concentrator/raw/version.php?ref=main');
 define('UPSTREAM_VERSION_CACHE', __DIR__ . '/../upstream_version.cache');
 define('UPSTREAM_VERSION_TTL', 3600); // seconds
+define('UPSTREAM_VERSION_FAIL_TTL', 900); // seconds — how long to cache a failed check

 /**
 * Fetch the upstream version string from the git repo, with a 1-hour file cache.
 * Returns null silently on any failure so the page always loads.
+ *
+ * Failures are cached briefly too, so an unreachable git server does not
+ * add a cURL wait to every dashboard load (the page auto-refreshes every 30 s).
 */
 function fetch_upstream_version(): ?string {
-    // Return cached value if still fresh
+    // Return cached value (success or failure) if still fresh
    if (file_exists(UPSTREAM_VERSION_CACHE)) {
        $cached = json_decode(file_get_contents(UPSTREAM_VERSION_CACHE), true);
-        if (isset($cached['version'], $cached['checked_at'])
-            && (time() - $cached['checked_at']) < UPSTREAM_VERSION_TTL) {
-            return $cached['version'];
+        if (isset($cached['checked_at'])) {
+            $ttl = isset($cached['version']) ? UPSTREAM_VERSION_TTL : UPSTREAM_VERSION_FAIL_TTL;
+            if ((time() - $cached['checked_at']) < $ttl) {
+                return $cached['version'] ?? null;
+            }
        }
    }

@@ -271,27 +318,30 @@ function fetch_upstream_version(): ?string {
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 5,
+        CURLOPT_CONNECTTIMEOUT => 2,
        CURLOPT_FOLLOWLOCATION => true,
    ]);
    $body = curl_exec($ch);
    $code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);

-    if ($body === false || $code !== 200) {
-        return null;
-    }
-
-    if (!preg_match("/define\s*\(\s*'APP_VERSION'\s*,\s*'([^']+)'\s*\)/", $body, $m)) {
-        return null;
-    }
-
-    $version = $m[1];
+    if ($body !== false && $code === 200
+        && preg_match("/define\s*\(\s*'APP_VERSION'\s*,\s*'([^']+)'\s*\)/", $body, $m)) {
        file_put_contents(
            UPSTREAM_VERSION_CACHE,
-        json_encode(['version' => $version, 'checked_at' => time()]),
+            json_encode(['version' => $m[1], 'checked_at' => time()]),
            LOCK_EX
        );
-    return $version;
+        return $m[1];
+    }
+
+    // Cache the failure so we don't retry on every page load
+    file_put_contents(
+        UPSTREAM_VERSION_CACHE,
+        json_encode(['version' => null, 'checked_at' => time()]),
+        LOCK_EX
+    );
+    return null;
 }

 /**
@@ -344,8 +394,6 @@ function global_stats(): array {
    $total = (int)$pdo->query('SELECT COUNT(*) FROM connections')->fetchColumn();

    $since = date('Y-m-d H:i:s', time() - RATE_WINDOW_SECONDS);
-    $recent = (int)$pdo->prepare('SELECT COUNT(*) FROM connections WHERE occurred_at >= ?')
-                        ->execute([$since]) ? : 0;
    $s = $pdo->prepare('SELECT COUNT(*) FROM connections WHERE occurred_at >= ?');
    $s->execute([$since]);
    $recent = (int)$s->fetchColumn();
@@ -27,7 +27,8 @@ CREATE TABLE IF NOT EXISTS connections (
    created_at  DATETIME        NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    KEY idx_node_occurred (node_id, occurred_at),
-    KEY idx_src_ip (src_ip),
+    KEY idx_occurred (occurred_at),
+    KEY idx_src_ip_occ (src_ip, occurred_at),
    KEY idx_dst_port (dst_port),
    CONSTRAINT fk_conn_node FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
@@ -41,6 +41,43 @@ if ($row > 0) {
    echo "Dropped legacy uq_event unique key.\n";
 }

+// Migration: add occurred_at index if upgrading from an older schema.
+// Time-window queries (dashboard, JSON API, purge) full-scanned the table
+// without it, so page loads slowed down as the table grew.
+$row = $pdo->query(
+    "SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME   = 'connections'
+       AND INDEX_NAME   = 'idx_occurred'"
+)->fetchColumn();
+if ($row == 0) {
+    $pdo->exec("ALTER TABLE connections ADD KEY idx_occurred (occurred_at)");
+    echo "Added idx_occurred index.\n";
+}
+
+// Migration: replace the standalone src_ip index with (src_ip, occurred_at)
+// so frequent_ips() resolves as a covering index scan.
+$row = $pdo->query(
+    "SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME   = 'connections'
+       AND INDEX_NAME   = 'idx_src_ip_occ'"
+)->fetchColumn();
+if ($row == 0) {
+    $pdo->exec("ALTER TABLE connections ADD KEY idx_src_ip_occ (src_ip, occurred_at)");
+    echo "Added idx_src_ip_occ index.\n";
+}
+$row = $pdo->query(
+    "SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME   = 'connections'
+       AND INDEX_NAME   = 'idx_src_ip'"
+)->fetchColumn();
+if ($row > 0) {
+    $pdo->exec("ALTER TABLE connections DROP INDEX idx_src_ip");
+    echo "Dropped redundant idx_src_ip index.\n";
+}
+
 // Migration: create settings table and seed default retention if upgrading
 $pdo->exec(
    "CREATE TABLE IF NOT EXISTS settings (
@@ -0,0 +1,154 @@
+-- ============================================================================
+-- portspoof_concentrator — database update to v2610.1
+--
+-- Brings an existing database schema up to v2610.1. Safe to run from any
+-- previous version and safe to run more than once: every step is idempotent
+-- and prints a status line for what it did.
+--
+-- Usage (note the database name — the script uses DATABASE() in its checks):
+--
+--   mysql -u portspoof -p portspoof_concentrator < "update(2610.1).sql"
+--
+-- Note: on a large connections table the ADD KEY steps take time proportional
+-- to the table size (online DDL, but they scan all existing rows). Consider
+-- running during a quiet window.
+-- ============================================================================
+
+-- ----------------------------------------------------------------------------
+-- 1. Base tables — no-op if they already exist (a brand-new database is
+--    brought fully up to date by this file alone)
+-- ----------------------------------------------------------------------------
+
+CREATE TABLE IF NOT EXISTS nodes (
+    id          INT UNSIGNED    NOT NULL AUTO_INCREMENT,
+    name        VARCHAR(100)    NOT NULL,
+    api_url     VARCHAR(255)    NOT NULL COMMENT 'e.g. https://192.168.1.10:8080',
+    username    VARCHAR(100)    NOT NULL DEFAULT 'admin',
+    password    VARCHAR(255)    NOT NULL,
+    verify_ssl  TINYINT(1)      NOT NULL DEFAULT 0,
+    enabled     TINYINT(1)      NOT NULL DEFAULT 1,
+    last_fetched_at DATETIME    NULL,
+    last_event_at   DATETIME(6) NULL COMMENT 'timestamp of the newest ingested event from this node',
+    created_at  DATETIME        NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id),
+    UNIQUE KEY uq_api_url (api_url)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
+CREATE TABLE IF NOT EXISTS connections (
+    id          BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
+    node_id     INT UNSIGNED    NOT NULL,
+    occurred_at DATETIME(6)     NOT NULL COMMENT 'timestamp from portspoof_py',
+    src_ip      VARCHAR(45)     NOT NULL,
+    src_port    SMALLINT UNSIGNED NOT NULL,
+    dst_port    SMALLINT UNSIGNED NOT NULL,
+    banner_hex  TEXT            NULL,
+    banner_len  INT UNSIGNED    NOT NULL DEFAULT 0,
+    created_at  DATETIME        NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (id),
+    KEY idx_node_occurred (node_id, occurred_at),
+    KEY idx_occurred (occurred_at),
+    KEY idx_src_ip_occ (src_ip, occurred_at),
+    KEY idx_dst_port (dst_port),
+    CONSTRAINT fk_conn_node FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
+CREATE TABLE IF NOT EXISTS settings (
+    key_name    VARCHAR(64)  NOT NULL,
+    value       TEXT         NOT NULL,
+    PRIMARY KEY (key_name)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
+INSERT IGNORE INTO settings (key_name, value) VALUES ('retention_days', '7');
+INSERT IGNORE INTO settings (key_name, value) VALUES ('frequent_ip_threshold', '5');
+
+-- ----------------------------------------------------------------------------
+-- 2. v2603.x migration: nodes.last_event_at (the per-node deduplication
+--    cursor). No-op for databases created with the current schema.
+-- ----------------------------------------------------------------------------
+
+SET @sql = IF(
+    (SELECT COUNT(*) FROM information_schema.COLUMNS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME = 'nodes'
+       AND COLUMN_NAME = 'last_event_at') = 0,
+    'ALTER TABLE nodes ADD COLUMN last_event_at DATETIME(6) NULL COMMENT ''timestamp of the newest ingested event from this node'' AFTER last_fetched_at',
+    'SELECT ''nodes.last_event_at: already present, skipped'' AS status'
+);
+PREPARE stmt FROM @sql;
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+-- ----------------------------------------------------------------------------
+-- 3. v2603.x migration: drop the legacy uq_event unique key
+-- ----------------------------------------------------------------------------
+
+SET @sql = IF(
+    (SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME = 'connections'
+       AND INDEX_NAME = 'uq_event') > 0,
+    'ALTER TABLE connections DROP INDEX uq_event',
+    'SELECT ''connections.uq_event: not present, skipped'' AS status'
+);
+PREPARE stmt FROM @sql;
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+-- ----------------------------------------------------------------------------
+-- 4. v2610.1: idx_occurred — lets the time-window queries (dashboard stats,
+--    recent connections, JSON API, purge) use an index instead of scanning
+--    the whole connections table
+-- ----------------------------------------------------------------------------
+
+SET @sql = IF(
+    (SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME = 'connections'
+       AND INDEX_NAME = 'idx_occurred') = 0,
+    'ALTER TABLE connections ADD KEY idx_occurred (occurred_at)',
+    'SELECT ''connections.idx_occurred: already present, skipped'' AS status'
+);
+PREPARE stmt FROM @sql;
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+-- ----------------------------------------------------------------------------
+-- 5. v2610.1: idx_src_ip_occ — makes the frequent_ips() GROUP BY a covering
+--    index scan. Must run before step 6 so src_ip coverage is never lost.
+-- ----------------------------------------------------------------------------
+
+SET @sql = IF(
+    (SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME = 'connections'
+       AND INDEX_NAME = 'idx_src_ip_occ') = 0,
+    'ALTER TABLE connections ADD KEY idx_src_ip_occ (src_ip, occurred_at)',
+    'SELECT ''connections.idx_src_ip_occ: already present, skipped'' AS status'
+);
+PREPARE stmt FROM @sql;
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+-- ----------------------------------------------------------------------------
+-- 6. v2610.1: drop the now-redundant standalone idx_src_ip (fully covered
+--    by idx_src_ip_occ) — one less index to maintain on every insert
+-- ----------------------------------------------------------------------------
+
+SET @sql = IF(
+    (SELECT COUNT(*) FROM information_schema.STATISTICS
+     WHERE TABLE_SCHEMA = DATABASE()
+       AND TABLE_NAME = 'connections'
+       AND INDEX_NAME = 'idx_src_ip') > 0,
+    'ALTER TABLE connections DROP INDEX idx_src_ip',
+    'SELECT ''connections.idx_src_ip: not present, skipped'' AS status'
+);
+PREPARE stmt FROM @sql;
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+-- ============================================================================
+-- Done. Verify with:
+--   SHOW INDEX FROM connections;
+-- Expected indexes: PRIMARY, idx_node_occurred, idx_occurred,
+--                   idx_src_ip_occ, idx_dst_port
+-- ============================================================================
@@ -1,2 +1,2 @@
 <?php
-define('APP_VERSION', '2603.18');
+define('APP_VERSION', '2610.1');
Author	SHA1	Message	Date
daprogs	d382c77995	docs: add SQL-only schema update script for v2610.1 update(2610.1).sql brings an existing database up to the current schema without PHP: base tables (CREATE IF NOT EXISTS), the v2603.x migrations (last_event_at column, uq_event removal) and the v2610.1 index changes (idx_occurred, idx_src_ip_occ, drop idx_src_ip). All steps are idempotent via information_schema checks + PREPARE/EXECUTE, so the file is safe to re-run and portable across MySQL 8 and MariaDB.	2026-10-19 12:12:00 -04:00
daprogs	5649930060	release: bump version to 2610.1	2026-10-18 23:43:03 -04:00
daprogs	b479f92a63	perf: delete purged connections in batches of 5000 The purge ran one DELETE over the whole retention cutoff in a single transaction, spiking undo/binlog and holding row locks on a table the fetch cron writes to. Now it loops in batches of 5000 rows with a short pause between batches, keeping each commit small.	2026-10-18 22:18:08 -04:00
daprogs	4ce1502831	fix: drop duplicate count query; cache upstream version failures - global_stats() executed the 'last N seconds' COUNT twice per page load (prepare()->execute() result was discarded, then re-run) - fetch_upstream_version() cached nothing on failure, so an unreachable git server added a cURL wait (up to 5 s) to every dashboard load via the 30 s auto-refresh; failures are now cached for 15 minutes and get a 2 s connect timeout	2026-10-18 22:12:18 -04:00
daprogs	b0fdd99ad6	perf: batch connection inserts in ingest_connections Replace one INSERT per event (up to FETCH_LIMIT auto-committed round trips per node per poll) with multi-row INSERTs in chunks of 200, wrapped in a single transaction per node. Same data, one commit instead of hundreds, which was making fetch runs slower as the table grew.	2026-10-18 22:01:03 -04:00
daprogs	849360e61c	perf: add occurred_at indexes to connections Time-window queries (dashboard stats, recent connections, JSON API, purge) had no usable index and full-scanned the connections table on every load, so the dashboard slowed down as data accumulated. - Add idx_occurred (occurred_at) for counts, MAX, ORDER BY ... LIMIT, and the purge delete - Add idx_src_ip_occ (src_ip, occurred_at) so frequent_ips() runs as a covering index scan - Drop the now-redundant standalone idx_src_ip Migrations in setup.php are idempotent (information_schema checks) and portable across MySQL 8 and MariaDB.	2026-10-18 21:56:33 -04:00