================================================================================ csweb-gui -- Frontend / Backend Interaction Behaviours behaviours.txt See also: paths.txt (project structure), guideline.txt (session rules) ================================================================================ ================================================================================ 1. COMMAND FLOW OVERVIEW ================================================================================ Browser -> admin.pl (CGI on frontend Apache) -> &exe("command") -> socket($cmd, $ip) [admin-wlib.pl] -> TCP port 45000 -> server.pl (backend, managed server) -> executes locally, returns output ALL OS commands MUST go through &exe(). Never use system(), backticks, or local file ops for backend operations. Menu code runs on the frontend; the backend is a separate remote server. ================================================================================ 2. COMMAND ENCODING (b64cmd) ================================================================================ Problem: commands may contain ::, newlines, shell special chars, quotes. All break the :: delimiter protocol. Solution: socket() in admin-wlib.pl base64-encodes every non-status command: $cmd = "b64cmd:" . encode_base64($input, "") . $par_options; Exceptions (sent plain -- short ASCII, no special chars possible): hostname, hn, sc, rh, on, hi, os, dn, ip, fn, alive, stat, mem server.pl decodes in process_command() after options are stripped: if ($q =~ s/^b64cmd://) { $q = decode_base64($q); } else { $q =~ s//\n/gs; } # legacy fallback for old frontend IMPORTANT: b64cmd wraps the ENTIRE command including the ~ refresh marker. After decode, server.pl must re-apply ~ strip (see section 8). ================================================================================ 3. OPTIONS BLOCK (add_cmd_opt) ================================================================================ Every command gets an options block appended by add_cmd_opt(): "b64cmd:BASE64 (::line::member::upsrv::w2::action::opt::filename::data::cache::time::cfn::auth::upauth::)" - Appended AFTER the base64-encoded command (outside the base64) - Parsed by parameter() in server.pl -> fills %data hash - Stripped: $q =~ s/ *\(::.*//s; before command execution Key fields: line source line number (debugging) member member name~ip (e.g. cs_omni~192.168.2.203) upsrv upload server for callback path (empty = use calling IP) cfn cache filename base (ip_cmd_hash, max 55 chars) auth SHA hash shared secret (must match cfg/server.auth) cache "nocache" to bypass cache Auth is validated for all non-localhost connections. 127.0.0.1 is always trusted; auth check skipped. get_pw() is called before EVERY remote auth comparison (no restart needed when server.auth changes). ================================================================================ 4. COMMAND TERMINATOR ================================================================================ Every syswrite in socket() appends \n: $socket->syswrite($cmd . "\n"); server.pl read loop exits on: 1. $message =~ /:EoS:\n?$/s -- chunked upload end marker (priority) 2. $message =~ /\n$/s -- \n terminator (normal commands) 3. $n < $chunk_size -- fallback for old clients without \n Without \n, the blocking read loop would hang until alarm timeout. ================================================================================ 5. SOCKET MODES ================================================================================ Server socket: non-blocking (required for IO::Select) Accepted client sockets: blocking (->blocking(1) after accept) Why blocking for clients: - can_read() pre-checks data availability -> blocking sysread never hangs - \n terminator exits loop after last chunk (no second sysread attempt) - Avoids EAGAIN race on multi-chunk reads safe_write: uses IO::Select->can_write(30) per chunk for flow control. IO::Select object created ONCE per reply (not per chunk write). ================================================================================ 6. RESPONSE TRANSMISSION ================================================================================ End marker: server.pl appends ":EoS:\n" after all response data. admin-wlib.pl reads until ":EoS:\n", then strips it. Small (< CHUNK_SIZE = 16 KB): single syswrite "$reply\:EoS:\n" Large (>= CHUNK_SIZE): sent in chunks; :EoS:\n only if ALL chunks succeed. Partial write -> :EoS: NOT sent -> admin-wlib.pl detects incomplete transfer. :BoD: / :EoD: (legacy): Old server.pl versions wrapped responses in :BoD:...:EoD: markers. Current server.pl sends plain output; :EoS: is the only end marker. admin-wlib.pl still strips :BoD:/:EoD: if present (backward compat). Error if :BoD: present but :EoD: missing (incomplete transfer). ================================================================================ 7. DIRECT PATH vs CALLBACK PATH ================================================================================ Direct: $socket->syswrite($cmd . "\n"), response received inline. Callback triggers when: $current{'on'} =~ /callback/ OR value_upsrv != "" OR length($cmd) >= 4 KB OR $cmd =~ /^rcmd::/ Callback flow: 1. admin-wlib.pl writes $cmd to tmp/UPFILE.rcmd 2. If upsrv set: uploads via curl HTTPS to upsrv 3. Sends "rcmd::UPFILE (::options::)" via socket 4. server.pl downloads UPFILE, executes, uploads result as UPFILE.data 5. Returns filename; admin-wlib.pl downloads .data file File extensions: .rcmd (command), .tmp (deleted after get), .data (result) MAX_DIRECT_BYTES = 256 KB: Responses > 256 KB are uploaded via callback, not sent directly. CRITICAL: if a backend command generates > 256 KB intermediate output (e.g. scraping a large HTML page), it triggers the callback path and may fail if cs_connect.pl is unreachable. Use JSON APIs instead. ================================================================================ 8. ~ REFRESH MARKER (monitor.pl auto-cache) ================================================================================ Menu code appends ~ to &exe() commands to enable auto-refresh: &exe("zpool list~") -- monitor.pl re-executes every ~20s server.pl strips ~ BEFORE b64cmd decode (line 468): if ($q =~ /~$/) { $refresh = "~"; $q =~ s/~$//s; } PROBLEM: ~ is inside the base64 payload. The pre-decode strip fires on the b64cmd string (which doesn't end with ~). After decode, ~ reappears. FIX: server.pl re-strips ~ AFTER b64cmd decode too. After execution, .last cache file written as: "$timestamp\n$cmd$refresh\n" monitor.pl re-executes commands whose .last entry ends with ~. ~ must NEVER reach zfs/zpool/powershell: "zpool get all~" -> zpool: "bad property 'all~'" ================================================================================ 9. CACHE MECHANISM ================================================================================ Location: /$tf/csweb-gui/tmp/ IPPREFIX_cmdname_hash.data cached result (CACHE_TTL_SECS = 30s) IPPREFIX_cmdname_hash.last timestamp\noriginal_cmd cfn (cache filename base): c_filename() in admin-wlib.pl: ip + "_" + sanitized_cmd + "_" + MD5_hash Max 55 chars. Included in options block. Parsed as $data{'cfn'} by server.pl. Cache bypass: "nocache" in options cache field, or command "dc" (del all .data). GUARD: if $data{'cfn'} is empty (command without options block), server.pl generates fallback name from ip+command. Without guard: empty cfn -> filename=".data" -> stale unrelated file served. ================================================================================ 10. RCMD COMMANDS ================================================================================ rcmd:writefile::PATH::CONTENT::EndOfFile:: Writes file to PATH on backend. Content is b64cmd-decoded by the central decode before write_file() sees it. Plain text arrives at the handler. PATH must have a valid file extension. rcmd:appendfile::PATH::CONTENT::EndOfFile:: Appends to file. Same decoding as writefile. rcmd::FILENAME (callback) Tell server.pl to download command from FILENAME via curl. rcmd::unlock::FILESYSTEM::KEYFILE Unlock ZFS encrypted filesystem. rcmd:cs_update Trigger csweb-gui update. rcmd:del_tmp_keys Delete temporary key files. ENCODING RULE: menu code passes RAW content to &exe(). socket() b64cmd-encodes the entire command including file content. Never add , , or base64 in menu/lib code (s3lib.pl etc.). ================================================================================ 11. $tf vs $current{'tf'} -- PATH TOKENS ================================================================================ $tf (frontend): Set at admin.pl startup from $^O: Windows -> "xampp", Unix -> "var" Used ONLY for local frontend file ops: require, glob, open, -f tests. NEVER overwrite -- breaks local file loading. $wpath/$dpath/$tpath are all derived from $tf at startup. $current{'tf'} (backend): Set after &exe('on') returns backend OS: $current{'tf'} = ($current{'fn'} eq "windows") ? "xampp" : "var"; Used for ALL &exe() paths sent to the backend. Example: &exe("ls /$current{'tf'}/csweb-gui/cfg/s3/bin/") Windows frontend + OmniOS backend: $tf = "xampp" (frontend), $current{'tf'} = "var" (backend) Using $tf in &exe() -> "cannot access /xampp/..." on OmniOS. TEMPLATE FILES: read from frontend with $tf, write to backend with $current{'tf'}: open(my $fh, '<', "/$tf/csweb-gui/startup/...") or die; my $content = <$fh>; &exe("rcmd:writefile::/$current{'tf'}/csweb-gui/cfg/...::..."); Never &exe("cat $src") to read templates -- runs on backend, not frontend. ================================================================================ 12. OS DETECTION ================================================================================ &exe('on') returns: "family;variant;id;name;OS_string;zfs_version;cs_version;callback_flag" Stored in $current{'on'}. Derived by admin.pl: $current{'fn'}: "windows"|"illumos"|"linux"|"solaris"|"freebsd"|"macos"|... $current{'ext'}: ".exe" (Windows) or "" (Unix) $current{'scr'}: ".bat" (Windows) or ".sh" (Unix) $current{'tf'}: "xampp" or "var" callback flag: if $current{'on'} =~ /callback/ -> use callback for ALL commands. ALWAYS use $current{'scr'} and $current{'ext'} in menu code. Never hardcode .bat / .sh / .exe -- breaks cross-OS access. ================================================================================ 13. AUTH / SECURITY ================================================================================ Shared secret: Backend: cfg/server.auth (first line, whitespace stripped) Frontend: _log/group/hostname~ip.txt (first line, whitespace stripped) Both must match. Re-read on every remote auth check (no restart needed). cs_connect.pl security: Validates auth against cfg/server.auth before all operations. Input sanitization: getfile/upfile filtered to [a-zA-Z0-9._\-~:] Lockfile opened in append mode >> (creates if missing). unlink only after path traversal check (no ../ or / in filename). ================================================================================ 14. ERROR HANDLING ================================================================================ server.pl returns "error:: message" for failures. admin-wlib.pl detects "error::" and calls &mess() unless nomess mode or SMART/GitHub (which can legitimately contain "error:"). Common errors: "error:: EndOfFile not found" rcmd:writefile content had :: issue "error:: socket auth wrong" server.auth mismatch "datatransfer via upload error::" callback path file not found "socket read timeout after Ns" command took too long "Socketerror or backend not started" port 45000 not reachable ================================================================================ 15. MONITOR.PL CACHE SERVICE ================================================================================ Runs as persistent service with 20s sleep loop. Activity window: reads tmp/activity.log. If idle > 24s: loop pauses. .last file: "$timestamp\n$command\n" (with ~ if auto-refresh) .data file: cached command result Auto-refresh (command ends with ~): monitor.pl re-executes and updates .data every ~20s within activity window. Cleanup: .completekey -> deleted immediately .session older than 10h -> deleted .data older than 30s (no ~ marker) -> deleted On startup: ALL tmp files deleted except .last and .log. ================================================================================ 16. KNOWN GOTCHAS ================================================================================ 1. ~ in zfs property names ~ is inside b64cmd payload. Pre-decode strip misses it. After decode, "readonly~" reaches zfs -> "bad property 'readonly~'". Fix: server.pl re-strips ~ after b64cmd decode. 2. Empty cfn -> cache collision Command without options block -> $data{'cfn'}="" -> filename=".data" -> any stale .data file served. Fix: server.pl derives name from ip+cmd. 3. Local path (127.0.0.1) must use add_cmd_opt curl_auth() for non-curl commands returns input unchanged (no options). Fix: local path uses add_cmd_opt + b64cmd like remote path. 4. $current{'tf'} vs $tf (see section 11) Using $tf in &exe() paths for OmniOS backend -> /xampp/ not found. 5. Encoding belongs in socket(), not menu code Never add , , or base64 in s3lib.pl or other menu libs. socket() is the ONLY encoding point. 6. :: in file content breaks rcmd protocol socket() b64cmd-encodes the entire command; :: in content is safe inside base64. Without encoding, :: would split the rcmd fields incorrectly. 7. MAX_DIRECT_BYTES = 256 KB Backend commands generating >256 KB intermediate output trigger callback path. HTML page scraping can easily exceed this. Use JSON APIs. 8. :BoD: in output (old server.pl) Old versions wrapped ALL replies in :BoD:...:EoD:. admin-wlib.pl strips them; error if :BoD: present but :EoD: missing. 9. Line endings in startup scripts Windows .bat: CRLF (\r\n). Unix .sh: LF (\n) only. Always strip \r before encoding: $content =~ s/\r//g; 10. Template files on frontend, config on backend Startup scripts (.sh/.bat templates) exist only on the FRONTEND install. Read with Perl open() using $tf. Write to backend via rcmd:writefile using $current{'tf'}. Never &exe("cp $src $dst") for templates. ================================================================================ # ------------------------------------------------------------------------------- # RULE: auto.pl (monitor.pl) modifications # ------------------------------------------------------------------------------- # If auto.pl is modified, update the function comment block at the top: # # # -- Functions --------------------------------------------------------------- # # sub_name() -- short description # # other_sub() -- short description # # ------------------------------------------------------------------------------- # # Also update the $ver = "YY.MM.DD_HH:MM" and add a changelog entry. # Format matches the existing Description/Changelog block at the top of auto.pl. ================================================================================ GOLDEN RULE -- FILE PERMISSIONS ================================================================================ GOLDEN RULE: ALL file/dir/delete operations go through &exe() -> server.pl. WHY: The web CGI (admin.pl) runs as Apache user "nobody" -- no write/delete permissions outside the web root. server.pl runs as root/admin and can access all files. &exe("cmd") -- sends cmd to MEMBER SERVER ($current{"ip"}) &socket("cmd","ip") -- sends cmd to SPECIFIC IP (e.g. 127.0.0.1 = local) OPERATIONS ON MEMBER SERVER (remote, use &exe): &exe("cat $path") -- read file on backend &exe("mkdir -p $dir") -- create directory on backend &exe("chmod +x $file") -- set permissions on backend &exe("unlink $file") -- delete file via Perl unlink() on backend rcmd:writefile -- write file content on backend OPERATIONS ON FRONTEND SERVER (local, use &socket with 127.0.0.1): &socket("unlink $file","127.0.0.1") -- delete local file &socket("cat $path","127.0.0.1") -- read local file &socket("mkdir -p $dir","127.0.0.1") -- create local dir WHY NOT local Perl ops (open, unlink, -f, -d): CGI (admin.pl) runs as Apache "nobody" -- no write/delete rights. server.pl runs as root/admin -- full access on all platforms. unlink($file) in CGI = FAILS on Linux (nobody cannot delete root files) &socket("unlink $file","127.0.0.1") = works on Windows/Linux/OmniOS WRONG (nobody permissions): unlink($file) -- local Perl unlink as nobody open(my $fh, ">", $p) -- local write as nobody system("rm $file") -- shell as nobody `cat $file` -- backticks as nobody -f $file, -d $dir -- local stat as nobody NOTE: &exe("unlink") / &socket("unlink","127.0.0.1") preferred over rm -f because it works on all platforms including Windows. AUTHENTICATION: Every socket() call needs auth (shared secret from cfg/server.auth). Auth is stored in %cfg hash, keyed by IP: $cfg{$ip} -- auth for member server at $ip $cfg{"127.0.0.1"} -- auth for local frontend server socket() / exe() reads this automatically via add_cmd_opt(). For &socket() direct calls: auth is passed automatically if $cfg is loaded. NEVER hardcode auth values -- always use $cfg{$ip} or $cfg{"127.0.0.1"}. ================================================================================ CACHING ================================================================================ Cache is activated/deactivated via the top-level menu (interface.pl). Controlled by flag file: $wpath/cfg/cache_on (presence = cache active) HOW IT WORKS: admin-wlib.pl socket(): - Before every remote request: if (-f "$wpath/cfg/cache_on") -> compute cfn = ip + c_filename($input) (SHA256 of original command) -> check $wpath/tmp/$cfn.data + $cfn.last (timestamp) -> if age < CACHE_TTL_SECS (30s): return cached result, skip request - After every remote request: -> write result to $wpath/tmp/$cfn.data -> write timestamp + cmd to $cfn.last server.pl also checks its own cache (tmp/$cfn.data) before executing. Double-layer: frontend cache (admin-wlib) + backend cache (server.pl). CACHE KEY: cfn = c_filename($input) where $input is the ORIGINAL command BEFORE b64 encoding. cfn is passed in the options block (OUTSIDE base64) so server.pl has it. b64 encoding does NOT affect cache keys. FLAG FILE OPERATIONS (must go through server.pl -- CGI runs as nobody): Enable: &socket('rcmd:writefile::' . "$wpath/cfg/cache_on" . '::1::EndOfFile::','127.0.0.1') Disable: &socket("unlink $wpath/cfg/cache_on","127.0.0.1") Check: -f "$wpath/cfg/cache_on" (readable by nobody, local -f is OK here) Same pattern for log_on and debug_on flag files. BYPASS CACHE: append ~ to command: &exe("zpool list~") -> forces refresh, updates cache $zfs{'_cache'} = 'nocache' -> disables cache for this session ================================================================================ 17. SOCKET.LIB -- Shared Socket Communication Library ================================================================================ Location: data/menues/_lib/windows/socketlib.pl PURPOSE: Provides &exe(), &socket(), &add_cmd_opt(), &c_filename(), &load_group_auth() for any CGI script without pulling in the full admin-wlib.pl. Used by modifier.pl; can be required by other lightweight CGI scripts. USAGE: use vars qw($t $tf $wpath $tpath $debug %cfg %in %current %sys %zfs); require 'socket.lib'; &load_group_auth($in{'member'}); # fills %cfg, sets $current{'member_ip'} my $r = &exe("zpool list"); # sends to $current{'member_ip'} CACHE PARAMETER: &exe($cmd, undef, undef, $cache) $cache=1 allow (default), 0=nocache &socket($cmd,$ip,$to,$ln,$nm,$cache) Forwarded as "nocache" in options block if $cache=0 or cfg/cache_on absent. Double-layer: frontend check (socket.lib) + backend check (server.pl). REQUIRED VARS (caller must declare with use vars): $tf platform token ("xampp"|"var") $wpath /$tf/csweb-gui $tpath /$tf/csweb-gui/tmp $debug 0|1 %cfg auth keys keyed by IP (filled by load_group_auth) %current runtime state (member_ip, cmdopt, ...) %in CGI params (read by caller) %sys system state %zfs zfs cache ================================================================================ 18. MODIFIER.PL -- Form Field Modifier CGI ================================================================================ Location: data/wwwroot/cgi-bin/modifier.pl JS: data/wwwroot/_doc/modifier.js CSS: data/wwwroot/_doc/modifier.css EVERY FORM ELEMENT must submit (via modifier.js payload): id = $in{'id'} session ID ("username,authkey") member = $in{'member'} selected backend member ("hostname~ip") modifier.pl uses member to: 1. Extract IP from "hostname~ip" -> $current{'member_ip'} 2. Call &load_group_auth(member) -> loads only that member's key 3. &exe() sends commands to that member (not always 127.0.0.1) CACHE: cache=1 (default in JS payload): server may serve from cache (30s TTL) cache=0: "nocache" sent in options block; server always re-executes DEBUG LOG: Every request writes %in (all POST params) to: /$tf/csweb-gui/tmp/modifier.last Format: key => value, one per line (like print_hash in admin-wlib.pl) AUTH FLOW: 1. check_session(id): validates S_ID/S_IP/S_AGENT against session file Session file: $tpath/session.$REF (written by admin.pl getid_set) 2. &load_group_auth(member): reads _log/group/member.txt -> $cfg{$ip} Used by &exe() for socket auth to server.pl Note: no group-key auth check in modifier.pl itself -- session IS the auth. ================================================================================ 19. GET_ASYNC.PL -- Async Status Query CGI ================================================================================ Location: data/wwwroot/cgi-bin/get_async.pl PURPOSE: Read-only status checks (alive, success, load, health etc.) called directly by browser JS after page load. No session check -- sessions expire while page is open; group key from _log/group/member.txt is the auth. AUTH: load_group_auth(member) fills %cfg; &exe() uses this for socket auth. LOG: $tpath/get_async.log (append, max 10KB) Format per call: get_async YYYY.MM.DD_HH:MM member=... area=X subarea=Y property=Z value=... cache=1 repeat=0s session => S_ID_match=ok S_IP=... REMOTE=... [if session present] auth => member=... cfg_key_len=64 params => area/sub/prop host=... port=... cmd => curl -kIs https://... (or "(none: reason)") socket_error => ... (only if error) resp => {"ok":1,"color":"green",...} LOCK: $tpath/ga_area_sub_prop.lock (flock LOCK_NB) Prevents duplicate parallel calls for same check. REPEAT: repeat=0 (default): Content-Type application/json, one response, exit. repeat=N (seconds): Content-Type application/x-ndjson, loop max 1h, one JSON line per interval, sleep(N) between. DISPATCH: area / subarea / property (3-level) All commands via &exe() -> socket -> server.pl (same as admin.pl). $LAST_CMD tracked via $main::LAST_CMD set in socketlib.pl exe(). ================================================================================