NetAlertX/.github/copilot-instructions.md

NetAlertX AI Assistant Instructions

This is NetAlertX — network monitoring & alerting. NetAlertX provides Network inventory, awareness, insight, categorization, intruder and presence detection. This is a heavily community-driven project, welcoming of all contributions.

You are expected to be concise, opinionated, and biased toward security and simplicity.

Architecture (what runs where)

• Backend (Python): main loop + GraphQL/REST endpoints orchestrate scans, plugins, workflows, notifications, and JSON export.
  • Key: server/__main__.py, server/plugin.py, server/initialise.py, server/api_server/api_server_start.py
• Data (SQLite): persistent state in db/app.db; helpers in server/database.py and server/db/*.
• Frontend (Nginx + PHP + JS): UI reads JSON, triggers execution queue events.
  • Key: front/, front/js/common.js, front/php/server/*.php
• Plugins (Python): acquisition/enrichment/publishers under front/plugins/* with config.json manifests.
• Messaging/Workflows: server/messaging/*, server/workflows/*
• API JSON Cache for UI: generated under api/*.json

Backend loop phases (see server/__main__.py and server/plugin.py): once, schedule, always_after_scan, before_name_updates, on_new_device, on_notification, plus ad‑hoc run via execution queue. Plugins execute as scripts that write result logs for ingestion.

Plugin patterns that matter

• Manifest lives at front/plugins/<code_name>/config.json; code_name == folder, unique_prefix drives settings and filenames (e.g., ARPSCAN).
• Control via settings: <PREF>_RUN (phase), <PREF>_RUN_SCHD (cron-like), <PREF>_CMD (script path), <PREF>_RUN_TIMEOUT, <PREF>_WATCH (diff columns).
• Data contract: scripts write /tmp/log/plugins/last_result.<PREF>.log (pipe‑delimited: 9 required cols + optional 4). Use front/plugins/plugin_helper.py’s Plugin_Objects to sanitize text and normalize MACs, then write_result_file().
• Device import: define database_column_definitions when creating/updating devices; watched fields trigger notifications.

Standard Plugin Formats

• publisher: Sends notifications to services. Runs on_notification. Data source: self.
• dev scanner: Creates devices and manages online/offline status. Runs on schedule. Data source: self / SQLite DB.
• name discovery: Discovers device names via various protocols. Runs before_name_updates or on schedule. Data source: self.
• importer: Imports devices from another service. Runs on schedule. Data source: self / SQLite DB.
• system: Provides core system functionality. Runs on schedule or is always on. Data source: self / Template.
• other: Miscellaneous plugins. Runs at various times. Data source: self / Template.

Plugin logging & outputs

• Always check relevant logs first.

• Use logging as shown in other plugins.

• Collect results with Plugin_Objects.add_object(...) during processing and call plugin_objects.write_result_file() exactly once at the end of the script.

• Prefer to log a brief summary before writing (e.g., total objects added) to aid troubleshooting; keep logs concise at info level and use verbose or debug for extra context.

• Do not write ad‑hoc files for results; the only consumable output is last_result.<PREF>.log generated by Plugin_Objects.

API/Endpoints quick map

• Flask app: server/api_server/api_server_start.py exposes routes like /device/<mac>, /devices, /devices/export/{csv,json}, /devices/import, /devices/totals, /devices/by-status, plus nettools, events, sessions, dbquery, metrics, sync.
• Authorization: all routes expect header Authorization: Bearer <API_TOKEN> via get_setting_value('API_TOKEN').
• All responses need to return "success":<False:True> and if False an "error" message needs to be returned, e.g. {"success": False, "error": f"No stored open ports for Device"}

Conventions & helpers to reuse

• Settings: add/modify via ccd() in server/initialise.py or per‑plugin manifest. Never hardcode ports or secrets; use get_setting_value().
• Logging: use logger.mylog(level, [message]); levels: none/minimal/verbose/debug/trace.
• Time/MAC/strings: helper.py (timeNowDB, normalize_mac, sanitizers). Validate MACs before DB writes.
• DB helpers: prefer server/db/db_helper.py functions (e.g., get_table_json, device condition helpers) over raw SQL in new paths.

Dev workflow (devcontainer)

• Devcontainer philosophy: brutal simplicity. One user, everything writable, completely idempotent. No permission checks, no conditional logic, no sudo needed. If something doesn't work, tear down the wall and rebuild - don't patch. We unit test permissions in the hardened build.
• Permissions: Never chmod or chown during operations. Everything is already writable. If you need permissions, the devcontainer setup is broken - fix .devcontainer/scripts/setup.sh or .devcontainer/resources/devcontainer-Dockerfile instead.
• Files & Paths: Use environment variables (NETALERTX_DB, NETALERTX_LOG, etc.) everywhere. /data for persistent config/db, /tmp for runtime logs/api/nginx state. Never hardcode /data/db or relative paths.
• Database reset: Use the [Dev Container] Wipe and Regenerate Database task. Kills backend, deletes /data/{db,config}/*, runs first-time setup scripts. Clean slate, no questions.
• Services: use tasks to (re)start backend and nginx/PHP-FPM. Backend runs with debugpy on 5678; attach a Python debugger if needed.
• Run a plugin manually: python3 front/plugins/<code_name>/script.py (ensure sys.path includes /app/front/plugins and /app/server like the template).
• Testing: pytest available via Alpine packages. Tests live in test/; app code is under server/. PYTHONPATH is preconfigured to include workspace and /opt/venv site‑packages.
• Subprocess calls: ALWAYS set explicit timeouts. Default to 60s minimum unless plugin config specifies otherwise. Nested subprocess calls (e.g., plugins calling external tools) need their own timeout - outer plugin timeout won't save you.

What “done right” looks like

• When adding a plugin, start from front/plugins/__template, implement with plugin_helper, define manifest settings, and wire phase via <PREF>_RUN. Verify logs in /tmp/log/plugins/ and data in api/*.json.
• When introducing new config, define it once (core ccd() or plugin manifest) and read it via helpers everywhere.
• When exposing new server functionality, add endpoints in server/api_server/* and keep authorization consistent; update UI by reading/writing JSON cache rather than bypassing the pipeline.

Useful references

• Docs: docs/PLUGINS_DEV.md, docs/SETTINGS_SYSTEM.md, docs/API_*.md, docs/DEBUG_*.md
• Logs: All logs are under /tmp/log/. Plugin logs are very shortly under /tmp/log/plugins/ until picked up by the server.
  • plugin logs: /tmp/log/app.log
  • backend logs: /tmp/log/stdout.log and /tmp/log/stderr.log
  • frontend commands logs: /tmp/log/app_front.log
  • php errors: /tmp/log/app.php_errors.log
  • nginx logs: /tmp/log/nginx-access.log and /tmp/log/nginx-error.log

Assistant expectations:

• Be concise, opinionated, and biased toward security and simplicity.
• Reference concrete files/paths/environmental variables.
• Use existing helpers/settings.
• Offer a quick validation step (log line, API hit, or JSON export) for anything you add.
• Be blunt about risks and when you offer suggestions ensure they're also blunt,
• Ask for confirmation before making changes that run code or change multiple files.
• Make statements actionable and specific; propose exact edits.
• Request confirmation before applying changes that affect more than a single, clearly scoped line or file.
• Ask the user to debug something for an actionable value if you're unsure.
• Be sure to offer choices when appropriate.
• Always understand the intent of the user's request and undo/redo as needed.
• Above all, use the simplest possible code that meets the need so it can be easily audited and maintained.
• Always leave logging enabled. If there is a possiblity it will be difficult to debug with current logging, add more logging.
• Always run the testFailure tool before executing any tests to gather current failure information and avoid redundant runs.
• Always prioritize using the appropriate tools in the environment first. As an example if a test is failing use testFailure then runTests. Never runTests first.
• Docker tests take an extremely long time to run. Avoid changes to docker or tests until you've examined the exisiting testFailures and runTests results.
• Environment tools are designed specifically for your use in this project and running them in this order will give you the best results.