PNET Labs IOL console / Session crashing / clear Hung Telnet sessions

Category >

IOL node crashes and it looks like either we need to kill the process running IOL or clear the telnet connection queue as telnet connection drops

Sometimes we see ^M^M^M^M^M for any input made on telnet command line

telnet 127.0.0.1 30011

if telnet is not working then try

nc 127.0.0.1 30011

For Telnet, exit with:

Ctrl+]
quit

In case node is alive but not accepting the telnet console connections

For a listening TCP socket, Recv-Q is the current accept queue and Send-Q is its maximum backlog. Therefore, console port 30011 has 4 pending connections out of a maximum of 5. The IOL process is running, but it is not properly accepting or clearing console connections.

That explains why closing and reopening the console makes it worse and why you see buffered carriage returns as:

^M^M^M^M^M

Then show every connection, including the queued connections:

ss -antop '( sport = :30011 )'

Now forcibly close the established TCP console sockets without killing the IOL node:

ss -K state established '( sport = :30011 )'

The -K option asks the kernel to forcibly close matching sockets.

Wait one second and check again:

sleep 1
ss -antp '( sport = :30011 )'
ss -lntp '( sport = :30011 )'

The healthy listening result should be:

LISTEN 0 5 *:30011

Once Recv-Q returns to 0, connect locally from PNETLab:

telnet 127.0.0.1 30011

If local Telnet also shows ^M

The IOL console handler itself is wedged even though the IOS process remains alive. In that case the node must be restarted.

Restart only node 11

sleep 3

pgrep -af '^/opt/unetlab/wrappers/iol_wrapper .* -S 11 '
pgrep -af '^/opt/unetlab/tmp/1/11/x86_64_crb'
ss -lntp '( sport = :30011 )'

If node is not alive then we need to kill the PID Process

Take a note of the node ID such as 11 in this case , to list the process id

cd /opt/unetlab/tmp/1/1

fuser -vm /opt/unetlab/tmp/1/1
lsof +D /opt/unetlab/tmp/1/1 2>/dev/null

If an IOL process remains after the node has been stopped, terminate only that PID:

kill <PID>
sleep 2
kill -9 <PID>

Also check for leftover IOL processes:

pgrep -af 'i86bi|x86_64_crb|iou|iol'

Killing and Listing the sessions manually

-------------------List the stuck sessions--------------------
root@pnetlab:~# ss -lntp | grep ':30011'
LISTEN  4        5                            *:30011                  *:*       users:(("x86_64_crb_linu",pid=45801,fd=8),("sh",pid=45800,fd=8),("iol_wrapper",pid=45799,fd=8),("iol_wrapper",pid=45798,fd=8))

-------------------x--------------------

root@pnetlab:~# ss -antop '( sport = :30011 )'
State             Recv-Q             Send-Q                                   Local Address:Port                                    Peer Address:Port             
LISTEN            0                  5                                                    *:30011                                              *:*                 users:(("x86_64_crb_linu",pid=45801,fd=8),("sh",pid=45800,fd=8),("iol_wrapper",pid=45799,fd=8),("iol_wrapper",pid=45798,fd=8))
ESTAB             0                  0                               [::ffff:192.168.0.219]:30011                         [::ffff:192.168.0.151]:59433             users:(("iol_wrapper",pid=45798,fd=33))

-------------------Manually Kill--------------------

root@pnetlab:~# ss -K state established '( sport = :30011 )'
Netid            Recv-Q             Send-Q                                   Local Address:Port                                    Peer Address:Port
tcp              0                  0                               [::ffff:192.168.0.219]:30011                         [::ffff:192.168.0.151]:59433

-------------------List again and check--------------------

root@pnetlab:~# ss -antp '( sport = :30011 )'
State               Recv-Q                Send-Q                                Local Address:Port                                Peer Address:Port
LISTEN              0                     5                                                 *:30011                                          *:*                   users:(("x86_64_crb_linu",pid=45801,fd=8),("sh",pid=45800,fd=8),("iol_wrapper",pid=45799,fd=8),("iol_wrapper",pid=45798,fd=8))

-------------------x--------------------

root@pnetlab:~# ss -lntp '( sport = :30011 )'
State               Recv-Q                Send-Q                                Local Address:Port                                Peer Address:Port
LISTEN              0                     5                                                 *:30011                                          *:*                   users:(("x86_64_crb_linu",pid=45801,fd=8),("sh",pid=45800,fd=8),("iol_wrapper",pid=45799,fd=8),("iol_wrapper",pid=45798,fd=8))

Create a menu entry in the PNET GUI to trigger or clear the stuck telnet queue for nodes

This issue can happen to any node and not just IOL

Log in to the PNETLab server as root and paste this entire block:

cat > /root/install-pnetlab-clear-console.sh <<'INSTALL'
#!/usr/bin/env bash

set -euo pipefail

###############################################################################
# PNETLab - Clear Hung Console
#
# Creates:
#   /usr/local/sbin/pnetlab-clear-hung-console
#   /opt/unetlab/html/clear-hung-console.php
#   /opt/unetlab/html/themes/default/js/clear-hung-console.js
#   /etc/sudoers.d/pnetlab-clear-hung-console
#
# Also inserts a loader for clear-hung-console.js into the PNETLab page.
###############################################################################

ROOT="/opt/unetlab/html"

WRAPPER="/usr/local/sbin/pnetlab-clear-hung-console"
PHP_ENDPOINT="$ROOT/clear-hung-console.php"
CUSTOM_JS="$ROOT/themes/default/js/clear-hung-console.js"
SUDOERS_FILE="/etc/sudoers.d/pnetlab-clear-hung-console"

STAMP="$(date +%Y%m%d%H%M%S)"
BACKUP_DIR="/root/pnetlab-clear-console-backup-${STAMP}"

mkdir -p "$BACKUP_DIR"

###############################################################################
# Basic checks
###############################################################################

if [[ "$(id -u)" -ne 0 ]]; then
    echo "ERROR: Run this installer as root."
    exit 1
fi

for CMD in php python3 sudo visudo ss; do
    if ! command -v "$CMD" >/dev/null 2>&1; then
        echo "ERROR: Required command not found: $CMD"
        exit 1
    fi
done

if [[ ! -d "$ROOT" ]]; then
    echo "ERROR: PNETLab web directory not found:"
    echo "  $ROOT"
    exit 1
fi

mkdir -p "$(dirname "$CUSTOM_JS")"

###############################################################################
# Backup helper
###############################################################################

backup_file()
{
    local FILE="$1"

    if [[ ! -f "$FILE" ]]; then
        return
    fi

    local DEST="$BACKUP_DIR/${FILE#/}"

    mkdir -p "$(dirname "$DEST")"
    cp -a "$FILE" "$DEST"
}

backup_file "$WRAPPER"
backup_file "$PHP_ENDPOINT"
backup_file "$CUSTOM_JS"
backup_file "$SUDOERS_FILE"

###############################################################################
# Detect PHP/Apache web account
###############################################################################

WEB_USER="$(
    ps -eo user=,comm= |
    awk '
        $1 != "root" &&
        (
            $2 ~ /^php-fpm/ ||
            $2 == "apache2" ||
            $2 == "nginx"
        )
        {
            print $1
            exit
        }
    '
)"

if [[ -z "${WEB_USER}" ]] && id www-data >/dev/null 2>&1; then
    WEB_USER="www-data"
fi

if [[ -z "${WEB_USER}" ]]; then
    echo "ERROR: Could not determine the web/PHP user."
    exit 1
fi

if ! id "$WEB_USER" >/dev/null 2>&1; then
    echo "ERROR: Web account does not exist: $WEB_USER"
    exit 1
fi

echo "Detected PNETLab web account: $WEB_USER"

###############################################################################
# Create root wrapper
###############################################################################

cat > "$WRAPPER" <<'WRAPPER'
#!/usr/bin/env bash

set -euo pipefail

if [[ $# -ne 2 ]]; then
    echo "Usage: $0 NODE_ID CONSOLE_PORT" >&2
    exit 64
fi

NODE_ID="$1"
PORT="$2"

###############################################################################
# Validate arguments
###############################################################################

if [[ ! "$NODE_ID" =~ ^[0-9]+$ ]]; then
    echo "Invalid node ID: $NODE_ID" >&2
    exit 64
fi

if [[ ! "$PORT" =~ ^[0-9]+$ ]]; then
    echo "Invalid console port: $PORT" >&2
    exit 64
fi

NODE_ID=$((10#$NODE_ID))
PORT=$((10#$PORT))

if (( NODE_ID < 1 || NODE_ID > 99999 )); then
    echo "Node ID outside allowed range: $NODE_ID" >&2
    exit 64
fi

/*
 * Placeholder removed below.
 */
WRAPPER

python3 - <<'PY'
from pathlib import Path

path = Path("/usr/local/sbin/pnetlab-clear-hung-console")
text = path.read_text()

placeholder = '''/*
 * Placeholder removed below.
 */
'''

replacement = r'''if (( PORT < 30000 || PORT > 65535 )); then
    echo "Console port outside allowed range: $PORT" >&2
    exit 64
fi

SS_BIN="$(command -v ss)"

if [[ -z "$SS_BIN" || ! -x "$SS_BIN" ]]; then
    echo "Unable to locate ss." >&2
    exit 69
fi

echo "Node ID:      $NODE_ID"
echo "Console port: $PORT"
echo "Clearing established console connection..."

"$SS_BIN" -K state established "( sport = :${PORT} )"

echo "Console connection cleared for node $NODE_ID on port $PORT."
'''

if placeholder not in text:
    raise SystemExit("Wrapper placeholder not found")

path.write_text(text.replace(placeholder, replacement))
PY

chown root:root "$WRAPPER"
chmod 0755 "$WRAPPER"

bash -n "$WRAPPER"

###############################################################################
# Create sudoers permission
###############################################################################

cat > "$SUDOERS_FILE" <<EOF
$WEB_USER ALL=(root) NOPASSWD: $WRAPPER *
EOF

chown root:root "$SUDOERS_FILE"
chmod 0440 "$SUDOERS_FILE"

/usr/sbin/visudo -cf "$SUDOERS_FILE"

###############################################################################
# Create PHP backend
###############################################################################

cat > "$PHP_ENDPOINT" <<'PHP'
<?php

header('Content-Type: application/json; charset=utf-8');
header('Cache-Control: no-store, no-cache, must-revalidate');

function response_json($status, $data)
{
    http_response_code($status);

    echo json_encode(
        $data,
        JSON_UNESCAPED_SLASHES |
        JSON_UNESCAPED_UNICODE
    );

    exit;
}

###############################################################################
# POST only
###############################################################################

if (($_SERVER['REQUEST_METHOD'] ?? '') !== 'POST') {
    response_json(
        405,
        array(
            'success' => false,
            'message' => 'POST required.'
        )
    );
}

###############################################################################
# Require XMLHttpRequest header
###############################################################################

if (
    ($_SERVER['HTTP_X_REQUESTED_WITH'] ?? '')
    !==
    'XMLHttpRequest'
) {
    response_json(
        403,
        array(
            'success' => false,
            'message' => 'Invalid request.'
        )
    );
}

###############################################################################
# Same-origin check
###############################################################################

$requestHost = strtolower(
    preg_replace(
        '/:\d+$/',
        '',
        trim($_SERVER['HTTP_HOST'] ?? '')
    )
);

$source =
    $_SERVER['HTTP_ORIGIN']
    ??
    $_SERVER['HTTP_REFERER']
    ??
    '';

$sourceHost = strtolower(
    (string) parse_url(
        $source,
        PHP_URL_HOST
    )
);

if (
    $requestHost === '' ||
    $sourceHost === '' ||
    !hash_equals(
        $requestHost,
        $sourceHost
    )
) {
    response_json(
        403,
        array(
            'success' => false,
            'message' => 'Request origin rejected.'
        )
    );
}

###############################################################################
# Parse request body
###############################################################################

$rawBody = file_get_contents(
    'php://input'
);

$data = json_decode(
    $rawBody ?: '',
    true
);

if (!is_array($data)) {
    $data = $_POST;
}

$nodeValue =
    $data['node_id']
    ??
    null;

$portValue =
    $data['port']
    ??
    null;

###############################################################################
# Normalize node ID
###############################################################################

if (is_int($nodeValue)) {
    $nodeString =
        (string) $nodeValue;
}
elseif (is_string($nodeValue)) {
    $nodeString =
        trim($nodeValue);
}
else {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Missing node ID.'
        )
    );
}

###############################################################################
# Normalize port
###############################################################################

if (is_int($portValue)) {
    $portString =
        (string) $portValue;
}
elseif (is_string($portValue)) {
    $portString =
        trim($portValue);
}
else {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Missing console port.'
        )
    );
}

###############################################################################
# Validate node ID and port
###############################################################################

if (
    !preg_match(
        '/^[0-9]{1,5}$/',
        $nodeString
    )
) {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Invalid node ID.'
        )
    );
}

if (
    !preg_match(
        '/^[0-9]{1,5}$/',
        $portString
    )
) {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Invalid console port.'
        )
    );
}

$nodeId = (int) $nodeString;
$port   = (int) $portString;

if (
    $nodeId < 1 ||
    $nodeId > 99999
) {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Node ID outside allowed range.'
        )
    );
}

if (
    $port < 30000 ||
    $port > 65535
) {
    response_json(
        400,
        array(
            'success' => false,
            'message' => 'Console port outside allowed range.'
        )
    );
}

###############################################################################
# Execute fixed root wrapper
###############################################################################

$command =
    '/usr/bin/sudo -n ' .
    '/usr/local/sbin/pnetlab-clear-hung-console ' .
    escapeshellarg(
        (string) $nodeId
    ) .
    ' ' .
    escapeshellarg(
        (string) $port
    ) .
    ' 2>&1';

$output = array();
$returnCode = 1;

exec(
    $command,
    $output,
    $returnCode
);

if ($returnCode !== 0) {
    response_json(
        500,
        array(
            'success' => false,
            'message' =>
                'Unable to clear the console.',
            'details' =>
                implode("\n", $output)
        )
    );
}

response_json(
    200,
    array(
        'success' => true,
        'node_id' => $nodeId,
        'port' => $port,
        'message' => sprintf(
            'Hung console cleared for node %d on port %d.',
            $nodeId,
            $port
        )
    )
);
PHP

chown root:root "$PHP_ENDPOINT"
chmod 0644 "$PHP_ENDPOINT"

php -l "$PHP_ENDPOINT"

###############################################################################
# Create standalone JavaScript
###############################################################################

cat > "$CUSTOM_JS" <<'JAVASCRIPT'
(function () {
    'use strict';

    if (
        window
            .__pnetlabClearHungConsoleActualPortLoaded
    ) {
        return;
    }

    window
        .__pnetlabClearHungConsoleActualPortLoaded =
        true;

    ///////////////////////////////////////////////////////////////////////////
    // Display feedback
    ///////////////////////////////////////////////////////////////////////////

    function notifyUser(message, type) {
        if (
            typeof window.showLog
            ===
            'function'
        ) {
            window.showLog(
                message,
                type
            );

            return;
        }

        if (type === 'error') {
            window.alert(message);
        }
        else {
            console.log(message);
        }
    }

    ///////////////////////////////////////////////////////////////////////////
    // Obtain the ACTUAL console port from PNETLab
    ///////////////////////////////////////////////////////////////////////////

    function getActualConsolePort(nodeId) {
        try {
            if (
                !window.App ||
                !App.topology ||
                !App.topology.nodes ||
                !App.topology.nodes[nodeId] ||
                typeof
                    App.topology.nodes[nodeId].get
                    !==
                    'function'
            ) {
                return '';
            }

            var port =
                App.topology
                    .nodes[nodeId]
                    .get('port');

            port =
                String(
                    port || ''
                ).trim();

            if (
                !/^[0-9]+$/.test(port)
            ) {
                return '';
            }

            return port;
        }
        catch (error) {
            console.error(
                'Unable to read PNETLab console port',
                error
            );

            return '';
        }
    }

    ///////////////////////////////////////////////////////////////////////////
    // Add menu entry underneath Unlock
    ///////////////////////////////////////////////////////////////////////////

    function addMenuEntry() {
        var unlockActions =
            document.querySelectorAll(
                'a.action-nodeunlock'
            );

        Array.prototype.forEach.call(
            unlockActions,
            function (unlockAction) {

                var unlockLi =
                    unlockAction.closest(
                        'li'
                    );

                if (!unlockLi) {
                    return;
                }

                var menu =
                    unlockLi.parentElement;

                if (!menu) {
                    return;
                }

                if (
                    menu.querySelector(
                        'a.action-clearhungconsole'
                    )
                ) {
                    return;
                }

                var nodeId =
                    String(
                        unlockAction.getAttribute(
                            'data-path'
                        ) || ''
                    );

                var nodeName =
                    String(
                        unlockAction.getAttribute(
                            'data-name'
                        ) || ''
                    );

                if (
                    !/^[0-9]+$/.test(
                        nodeId
                    )
                ) {
                    return;
                }

                var li =
                    document.createElement(
                        'li'
                    );

                var action =
                    document.createElement(
                        'a'
                    );

                var icon =
                    document.createElement(
                        'i'
                    );

                action.className =
                    'action-clearhungconsole menu-manage';

                action.href =
                    'javascript:void(0)';

                action.setAttribute(
                    'data-path',
                    nodeId
                );

                action.setAttribute(
                    'data-name',
                    nodeName
                );

                icon.className =
                    'fa fa-refresh';

                action.appendChild(
                    icon
                );

                action.appendChild(
                    document.createTextNode(
                        ' Clear hung console'
                    )
                );

                li.appendChild(
                    action
                );

                unlockLi.insertAdjacentElement(
                    'afterend',
                    li
                );
            }
        );
    }

    ///////////////////////////////////////////////////////////////////////////
    // PNETLab creates the context menu dynamically
    ///////////////////////////////////////////////////////////////////////////

    function startObserver() {
        if (
            !document.documentElement
        ) {
            window.setTimeout(
                startObserver,
                100
            );

            return;
        }

        var observer =
            new MutationObserver(
                function () {
                    addMenuEntry();
                }
            );

        observer.observe(
            document.documentElement,
            {
                childList: true,
                subtree: true
            }
        );

        addMenuEntry();
    }

    startObserver();

    ///////////////////////////////////////////////////////////////////////////
    // Extra check each time a context menu is opened
    ///////////////////////////////////////////////////////////////////////////

    document.addEventListener(
        'contextmenu',
        function () {
            window.setTimeout(
                addMenuEntry,
                0
            );

            window.setTimeout(
                addMenuEntry,
                50
            );

            window.setTimeout(
                addMenuEntry,
                200
            );
        },
        true
    );

    ///////////////////////////////////////////////////////////////////////////
    // Handle click
    ///////////////////////////////////////////////////////////////////////////

    document.addEventListener(
        'click',
        function (event) {

            var target =
                event.target;

            if (
                !target ||
                typeof target.closest
                    !==
                    'function'
            ) {
                return;
            }

            var action =
                target.closest(
                    'a.action-clearhungconsole'
                );

            if (!action) {
                return;
            }

            event.preventDefault();
            event.stopPropagation();
            event.stopImmediatePropagation();

            var nodeId =
                String(
                    action.getAttribute(
                        'data-path'
                    ) || ''
                );

            var nodeName =
                String(
                    action.getAttribute(
                        'data-name'
                    ) || ''
                );

            if (
                !/^[0-9]+$/.test(
                    nodeId
                )
            ) {
                notifyUser(
                    'Invalid node ID.',
                    'error'
                );

                return;
            }

            /*
             * IMPORTANT:
             *
             * Do NOT calculate 30000 + NODE_ID.
             *
             * Read PNETLab's real assigned port.
             */
            var actualPort =
                getActualConsolePort(
                    nodeId
                );

            if (
                !actualPort
            ) {
                notifyUser(
                    'Unable to determine the actual console port.',
                    'error'
                );

                return;
            }

            var numericPort =
                Number(actualPort);

            if (
                numericPort < 30000 ||
                numericPort > 65535
            ) {
                notifyUser(
                    'Console port outside allowed range: ' +
                    actualPort,
                    'error'
                );

                return;
            }

            /*
             * Store it on the menu item too.
             * Useful for troubleshooting from DevTools.
             */
            action.setAttribute(
                'data-port',
                actualPort
            );

            var displayName =
                nodeName ||
                (
                    'node ' +
                    nodeId
                );

            if (
                !window.confirm(
                    'Clear the hung console for ' +
                    displayName +
                    '\n\n' +
                    'Actual console port: ' +
                    actualPort +
                    '?'
                )
            ) {
                return;
            }

            action.style.pointerEvents =
                'none';

            action.style.opacity =
                '0.5';

            fetch(
                '/clear-hung-console.php',
                {
                    method:
                        'POST',

                    credentials:
                        'same-origin',

                    headers: {
                        'Content-Type':
                            'application/json; charset=utf-8',

                        'X-Requested-With':
                            'XMLHttpRequest'
                    },

                    body:
                        JSON.stringify(
                            {
                                node_id:
                                    nodeId,

                                port:
                                    actualPort
                            }
                        )
                }
            )
            .then(
                function (response) {
                    return response
                        .text()
                        .then(
                            function (
                                responseText
                            ) {
                                var data;

                                try {
                                    data =
                                        JSON.parse(
                                            responseText
                                        );
                                }
                                catch (error) {
                                    data = {
                                        success:
                                            false,

                                        message:
                                            responseText
                                            ||
                                            'Invalid server response.'
                                    };
                                }

                                if (
                                    !response.ok ||
                                    !data.success
                                ) {
                                    var errorMessage =
                                        data.message
                                        ||
                                        'Unable to clear console.';

                                    if (
                                        data.details
                                    ) {
                                        errorMessage +=
                                            '\n' +
                                            data.details;
                                    }

                                    throw new Error(
                                        errorMessage
                                    );
                                }

                                return data;
                            }
                        );
                }
            )
            .then(
                function (data) {
                    notifyUser(
                        data.message,
                        'success'
                    );
                }
            )
            .catch(
                function (error) {
                    notifyUser(
                        error.message
                        ||
                        'Failed to clear console.',
                        'error'
                    );
                }
            )
            .then(
                function () {
                    action.style.pointerEvents =
                        '';

                    action.style.opacity =
                        '';
                }
            );
        },
        true
    );

    console.log(
        'PNETLab Clear Hung Console loaded - actual port mode'
    );
})();
JAVASCRIPT

chown root:root "$CUSTOM_JS"
chmod 0644 "$CUSTOM_JS"

###############################################################################
# Insert standalone JS loader into PNETLab
###############################################################################

export ROOT
export CUSTOM_JS
export STAMP
export BACKUP_DIR

python3 <<'PY'
import os
import re
import shutil
from pathlib import Path

root = Path(
    os.environ["ROOT"]
)

stamp = os.environ["STAMP"]

backup_root = Path(
    os.environ["BACKUP_DIR"]
)

loader_tag = (
    '<script src="/themes/default/js/'
    'clear-hung-console.js?v='
    + stamp +
    '"></script>'
)

###############################################################################
# Script tags
###############################################################################

script_pattern = re.compile(
    r'<script\b[^>]*>.*?</script>',
    re.IGNORECASE |
    re.DOTALL
)

###############################################################################
# Files/directories we do not want to modify
###############################################################################

skip_dirs = {
    "vendor",
    "node_modules",
    ".git",
}

skip_suffixes = {
    ".js",
    ".css",
    ".map",
    ".gz",
    ".br",
    ".png",
    ".jpg",
    ".jpeg",
    ".gif",
    ".ico",
    ".zip",
    ".tar",
    ".qcow2",
    ".bin",
}

patched = []

###############################################################################
# Search PNETLab web files
###############################################################################

for path in root.rglob("*"):

    if not path.is_file():
        continue

    if any(
        part in skip_dirs
        for part in path.parts
    ):
        continue

    if (
        path.suffix.lower()
        in
        skip_suffixes
    ):
        continue

    if ".bak" in path.name:
        continue

    try:
        if (
            path.stat().st_size
            >
            10_000_000
        ):
            continue

        raw =
            path.read_bytes()

    except OSError:
        continue

    if b"\x00" in raw:
        continue

    /*
     * Placeholder removed below.
     */
PY

python3 - <<'PY'
from pathlib import Path

p = Path("/root/install-pnetlab-clear-console.sh")
text = p.read_text()

placeholder = '''    /*
     * Placeholder removed below.
     */
PY
'''

replacement = r'''    try:
        text = raw.decode(
            "utf-8",
            errors="surrogateescape",
        )
    except UnicodeError:
        continue

    ###########################################################################
    # The page must load either the traditional actions.js or the React bundle.
    ###########################################################################

    if (
        "themes/default/js/actions.js"
        not in text
        and
        "store/public/react/js/main.js"
        not in text
    ):
        continue

    ###########################################################################
    # Remove an older copy of our loader, making installer idempotent.
    ###########################################################################

    def remove_old_loader(match):
        tag = match.group(0)

        if (
            "clear-hung-console.js"
            in tag
        ):
            return ""

        return tag

    cleaned =
        script_pattern.sub(
            remove_old_loader,
            text,
        )

    ###########################################################################
    # Find the best place to insert our loader.
    ###########################################################################

    matches =
        list(
            script_pattern.finditer(
                cleaned
            )
        )

    insertion_point = None

    # Prefer actions.js.
    for match in matches:
        if (
            "themes/default/js/actions.js"
            in match.group(0)
        ):
            insertion_point =
                match.end()

            break

    # Fall back to main React bundle.
    if insertion_point is None:
        for match in matches:
            if (
                "store/public/react/js/main.js"
                in match.group(0)
            ):
                insertion_point =
                    match.end()

                break

    if insertion_point is None:
        continue

    updated = (
        cleaned[:insertion_point]
        +
        "\n"
        +
        loader_tag
        +
        cleaned[insertion_point:]
    )

    ###########################################################################
    # Backup the page before modification
    ###########################################################################

    destination = (
        backup_root /
        str(path).lstrip("/")
    )

    destination.parent.mkdir(
        parents=True,
        exist_ok=True,
    )

    shutil.copy2(
        path,
        destination,
    )

    ###########################################################################
    # Save
    ###########################################################################

    path.write_text(
        updated,
        encoding="utf-8",
        errors="surrogateescape",
    )

    patched.append(
        str(path)
    )

###############################################################################
# Result
###############################################################################

if not patched:
    raise SystemExit(
        "ERROR: Could not locate a PNETLab page "
        "loading actions.js or main.js."
    )

print()
print("PNETLab page(s) patched:")

for item in patched:
    print(
        "  " + item
    )

print()
print(
    "JavaScript cache version: "
    +
    stamp
)
PY
'''

if placeholder not in text:
    raise SystemExit(
        "Installer placeholder not found."
    )

p.write_text(
    text.replace(
        placeholder,
        replacement,
    )
)
PY

###############################################################################
# Validate the finished installer itself
###############################################################################

echo
echo "Installer generated."
echo "Backup directory:"
echo "  $BACKUP_DIR"

###############################################################################
# Reload web services
###############################################################################

systemctl reload apache2 2>/dev/null || true
systemctl reload nginx 2>/dev/null || true

###############################################################################
# Final checks
###############################################################################

echo
echo "==============================================="
echo "Backend validation"
echo "==============================================="

php -l "$PHP_ENDPOINT"
bash -n "$WRAPPER"
/usr/sbin/visudo -cf "$SUDOERS_FILE"

echo
echo "==============================================="
echo "Installed files"
echo "==============================================="

ls -l \
    "$WRAPPER" \
    "$PHP_ENDPOINT" \
    "$CUSTOM_JS" \
    "$SUDOERS_FILE"

echo
echo "==============================================="
echo "JavaScript loader"
echo "==============================================="

grep -RIn \
    --exclude='*.js' \
    --exclude='*.bak*' \
    "clear-hung-console.js?v=${STAMP}" \
    "$ROOT" 2>/dev/null || true

echo
echo "==============================================="
echo "Installation completed successfully"
echo "==============================================="
echo
echo "Perform an Empty Cache and Hard Reload"
echo "in your browser before testing."
INSTALL

Now make sure the installer parses:

chmod 0700 /root/install-pnetlab-clear-console.sh
bash -n /root/install-pnetlab-clear-console.sh

Then run it:

/root/install-pnetlab-clear-console.sh

Browser refresh

After installation, in Chrome/Edge:

  1. Press F12.
  2. Go to Network.
  3. Tick Disable cache.
  4. Hold the Reload button.
  5. Choose Empty Cache and Hard Reload.

Verify the extension loaded

Browser console:

window.__pnetlabClearHungConsoleActualPortLoaded

Expected:

true

Check that the standalone script is loaded:

[...document.scripts]
    .filter(s =>
        s.src.includes('clear-hung-console.js')
    )
    .map(s => s.src)

You should see something such as:

https://PNETLAB/themes/default/js/clear-hung-console.js?v=20260808112800

Verify dynamic console-port handling

For example, your node 10 currently demonstrates exactly why we made this change:

App.topology.nodes["10"].get("port")

returns:

30014

The extension therefore sends:

{
    "node_id": "10",
    "port": "30014"
}

The backend executes:

/usr/local/sbin/pnetlab-clear-hung-console 10 30014

which runs:

ss -K state established '( sport = :30014 )'

Test the backend manually

First find a node’s actual port from the browser:

App.topology.nodes["10"].get("port")

Suppose it returns 30014.

On PNETLab:

ss -tnp state established '( sport = :30014 )'

Then manually run:

/usr/local/sbin/pnetlab-clear-hung-console 10 30014

Then:

ss -tnp state established '( sport = :30014 )'

The established console session should be gone.

Test PHP permissions

Find the web user:

WEB_USER="$(
    ps -eo user=,comm= |
    awk '$1 != "root" && ($2 ~ /^php-fpm/ || $2 == "apache2") {print $1; exit}'
)"

echo "$WEB_USER"

Then test:

sudo -u "$WEB_USER" \
sudo -n \
/usr/local/sbin/pnetlab-clear-hung-console \
10 30014

It should execute without asking for a password.

The important design advantage is that a node can be ID 10 while PNETLab assigns 30014, 30027, 30102, etc. Clear hung console follows whatever port PNETLab currently reports rather than assuming 30000 + node ID


Leave a Reply

Your email address will not be published. Required fields are marked *