diff --git a/.github/ISSUE_TEMPLATE/documentation-feedback.yml b/.github/ISSUE_TEMPLATE/documentation-feedback.yml index 5359a968..fac8b58a 100755 --- a/.github/ISSUE_TEMPLATE/documentation-feedback.yml +++ b/.github/ISSUE_TEMPLATE/documentation-feedback.yml @@ -14,7 +14,7 @@ body: label: What document or section does this relate to? description: | Please include a link to the file and section, if applicable. Be specific about what part of the documentation you are referencing. - placeholder: e.g. https://github.com/jokob-sk/NetAlertX/blob/main/docs/FRONTEND_DEVELOPMENT.md + placeholder: e.g. https://docs.netalertx.com/FRONTEND_DEVELOPMENT validations: required: true - type: textarea @@ -49,7 +49,7 @@ body: required: false - type: checkboxes attributes: - label: Can I help implement this? ๐Ÿ‘ฉโ€๐Ÿ’ป๐Ÿ‘จโ€๐Ÿ’ป + label: Can I help implement this? ๐Ÿ‘ฉโ€๐Ÿ’ป๐Ÿ‘จโ€๐Ÿ’ป description: The maintainer can provide guidance and review your changes. options: - label: "Yes, Iโ€™d like to help implement the improvement" diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 792f50cb..8ee485c3 100755 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -5,7 +5,7 @@ body: - type: checkboxes attributes: label: Is there an existing issue for this? - description: Please search to see if an open or closed issue already exists for the feature you are requesting. + description: Please search to see if an open or closed issue already exists for the feature you are requesting. options: - label: I have searched the existing open and closed issues required: true @@ -32,21 +32,21 @@ body: label: Anything else? description: | Links? References? Mockups? Anything that will give us more context about the feature you are encountering! - + Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. validations: required: true - type: checkboxes attributes: label: Am I willing to test this? ๐Ÿงช - description: I rely on the community to test unreleased features. If you are requesting a feature, please be willing to test it within 48h of test request. Otherwise, the feature might be pulled from the code base. + description: I rely on the community to test unreleased features. If you are requesting a feature, please be willing to test it within 48h of test request. Otherwise, the feature might be pulled from the code base. options: - label: I will do my best to test this feature on the `netlertx-dev` image when requested within 48h and report bugs to help deliver a great user experience for everyone and not to break existing installations. required: true - type: checkboxes attributes: - label: Can I help implement this? ๐Ÿ‘ฉโ€๐Ÿ’ป๐Ÿ‘จโ€๐Ÿ’ป - description: The maintainer will provide guidance and help. The implementer will read the PR guidelines https://jokob-sk.github.io/NetAlertX/DEV_ENV_SETUP/ + label: Can I help implement this? ๐Ÿ‘ฉโ€๐Ÿ’ป๐Ÿ‘จโ€๐Ÿ’ป + description: The maintainer will provide guidance and help. The implementer will read the PR guidelines https://docs.netalertx.com/DEV_ENV_SETUP/ options: - label: "Yes" - label: "No" diff --git a/.github/ISSUE_TEMPLATE/i-have-an-issue.yml b/.github/ISSUE_TEMPLATE/i-have-an-issue.yml index 62eb3821..63bbe77d 100755 --- a/.github/ISSUE_TEMPLATE/i-have-an-issue.yml +++ b/.github/ISSUE_TEMPLATE/i-have-an-issue.yml @@ -21,7 +21,7 @@ body: label: Is there an existing issue for this? description: Please search to see if an open or closed issue already exists for the bug you encountered. options: - - label: I have searched the existing open and closed issues and I checked the docs https://jokob-sk.github.io/NetAlertX/ + - label: I have searched the existing open and closed issues and I checked the docs https://docs.netalertx.com/ required: true - type: checkboxes attributes: @@ -87,7 +87,7 @@ body: PASTE LOG HERE. Using the triple backticks preserves format. ``` description: | - Logs with debug enabled (https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEBUG_TIPS.md) โš  + Logs with debug enabled (https://docs.netalertx.com/DEBUG_TIPS) โš  ***Generally speaking, all bug reports should have logs provided.*** Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. Additionally, any additional info? Screenshots? References? Anything that will give us more context about the issue you are encountering! diff --git a/.github/ISSUE_TEMPLATE/setup-help.yml b/.github/ISSUE_TEMPLATE/setup-help.yml index d1b54e20..44ac630e 100755 --- a/.github/ISSUE_TEMPLATE/setup-help.yml +++ b/.github/ISSUE_TEMPLATE/setup-help.yml @@ -21,11 +21,11 @@ body: label: Did I research? description: Please confirm you checked the usual places before opening a setup support request. options: - - label: I have searched the docs https://jokob-sk.github.io/NetAlertX/ + - label: I have searched the docs https://docs.netalertx.com/ required: true - label: I have searched the existing open and closed issues required: true - - label: I confirm my SCAN_SUBNETS is configured and tested as per https://github.com/jokob-sk/NetAlertX/blob/main/docs/SUBNETS.md + - label: I confirm my SCAN_SUBNETS is configured and tested as per https://docs.netalertx.com/SUBNETS required: true - type: checkboxes attributes: @@ -61,7 +61,7 @@ body: attributes: label: app.log description: | - Logs with debug enabled (https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEBUG_TIPS.md) โš  + Logs with debug enabled (https://docs.netalertx.com/DEBUG_TIPS) โš  ***Generally speaking, all bug reports should have logs provided.*** Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. Additionally, any additional info? Screenshots? References? Anything that will give us more context about the issue you are encountering! diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 9e18bea3..04a75ad6 100755 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -52,7 +52,7 @@ Backend loop phases (see `server/__main__.py` and `server/plugin.py`): `once`, ` ## Conventions & helpers to reuse - Settings: add/modify via `ccd()` in `server/initialise.py` or perโ€‘plugin manifest. Never hardcode ports or secrets; use `get_setting_value()`. -- Logging: use `mylog(level, [message])`; levels: none/minimal/verbose/debug/trace. `none` is used for most important messages that should always appear, such as exceptions. +- Logging: use `mylog(level, [message])`; levels: none/minimal/verbose/debug/trace. `none` is used for most important messages that should always appear, such as exceptions. Do NOT use `error` as level. - Time/MAC/strings: `server/utils/datetime_utils.py` (`timeNowDB`), `front/plugins/plugin_helper.py` (`normalize_mac`), `server/helper.py` (sanitizers). Validate MACs before DB writes. - DB helpers: prefer `server/db/db_helper.py` functions (e.g., `get_table_json`, device condition helpers) over raw SQL in new paths. @@ -71,13 +71,15 @@ Backend loop phases (see `server/__main__.py` and `server/plugin.py`): `once`, ` - When adding a plugin, start from `front/plugins/__template`, implement with `plugin_helper`, define manifest settings, and wire phase via `_RUN`. Verify logs in `/tmp/log/plugins/` and data in `api/*.json`. - When introducing new config, define it once (core `ccd()` or plugin manifest) and read it via helpers everywhere. - When exposing new server functionality, add endpoints in `server/api_server/*` and keep authorization consistent; update UI by reading/writing JSON cache rather than bypassing the pipeline. +- Always try following the DRY principle, do not re-implement functionality, but re-use existing methods where possible, or refactor to use a common method that is called multiple times +- If new functionality needs to be added, look at impenting it into existing handlers (e.g. `DeviceInstance` in `server/models/device_instance.py`) or create a new one if it makes sense. Do not access the DB from otehr application layers. +- Code files shoudln't be longer than 500 lines of code ## Useful references - Docs: `docs/PLUGINS_DEV.md`, `docs/SETTINGS_SYSTEM.md`, `docs/API_*.md`, `docs/DEBUG_*.md` - Logs: All logs are under `/tmp/log/`. Plugin logs are very shortly under `/tmp/log/plugins/` until picked up by the server. - plugin logs: `/tmp/log/plugins/*.log` - backend logs: `/tmp/log/stdout.log` and `/tmp/log/stderr.log` - - frontend commands logs: `/tmp/log/app_front.log` - php errors: `/tmp/log/app.php_errors.log` - nginx logs: `/tmp/log/nginx-access.log` and `/tmp/log/nginx-error.log` diff --git a/.github/workflows/mkdocs.yml b/.github/workflows/mkdocs.yml index 11ae0743..76a41896 100755 --- a/.github/workflows/mkdocs.yml +++ b/.github/workflows/mkdocs.yml @@ -3,7 +3,10 @@ name: Deploy MkDocs on: push: branches: - - main # Change if your default branch is different + - main + +permissions: + contents: write jobs: deploy: @@ -19,7 +22,15 @@ jobs: - name: Install MkDocs run: | - pip install mkdocs mkdocs-material && pip install mkdocs-github-admonitions-plugin + pip install mkdocs mkdocs-material + pip install mkdocs-github-admonitions-plugin + + - name: Build MkDocs + run: mkdocs build + + - name: Add CNAME + run: | + echo "docs.netalertx.com" > site/CNAME - name: Deploy MkDocs run: mkdocs gh-deploy --force diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 62a429ee..b47a067b 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,7 +12,7 @@ Please use the [GitHub Issue Tracker](https://github.com/jokob-sk/NetAlertX/issu - Documentation feedback ๐Ÿ“– Before opening a new issue: -- ๐Ÿ›‘ [Check Common Issues & Debug Tips](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEBUG_TIPS.md#common-issues) +- ๐Ÿ›‘ [Check Common Issues & Debug Tips](https://docs.netalertx.com/DEBUG_TIPS#common-issues) - ๐Ÿ” [Search Closed Issues](https://github.com/jokob-sk/NetAlertX/issues?q=is%3Aissue+is%3Aclosed) --- @@ -27,7 +27,7 @@ Please: - Follow existing **code style and structure** - Provide a clear title and description for your PR - If relevant, add or update tests and documentation -- For plugins, refer to the [Plugin Dev Guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS_DEV.md) +- For plugins, refer to the [Plugin Dev Guide](https://docs.netalertx.com/PLUGINS_DEV) --- @@ -47,7 +47,7 @@ By participating, you agree to follow our [Code of Conduct](./CODE_OF_CONDUCT.md ## ๐Ÿ“ฌ Contact -If you have more in-depth questions or want to discuss contributing in other ways, feel free to reach out at: +If you have more in-depth questions or want to discuss contributing in other ways, feel free to reach out at: ๐Ÿ“ง [jokob@duck.com](mailto:jokob@duck.com?subject=NetAlertX%20Contribution) We appreciate every contribution, big or small! ๐Ÿ’™ diff --git a/README.md b/README.md index e8798548..31832462 100755 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ # NetAlertX - Network, presence scanner and alert framework -Get visibility of what's going on on your WIFI/LAN network and enable presence detection of important devices. Schedule scans for devices, port changes and get alerts if unknown devices or changes are found. Write your own [Plugin](https://github.com/jokob-sk/NetAlertX/tree/main/docs/PLUGINS.md#readme) with auto-generated UI and in-build notification system. Build out and easily maintain your network source of truth (NSoT) and device inventory. +Get visibility of what's going on on your WIFI/LAN network and enable presence detection of important devices. Schedule scans for devices, port changes and get alerts if unknown devices or changes are found. Write your own [Plugin](https://docs.netalertx.com/PLUGINS#readme) with auto-generated UI and in-build notification system. Build out and easily maintain your network source of truth (NSoT) and device inventory. ## ๐Ÿ“‹ Table of Contents @@ -34,7 +34,7 @@ Get visibility of what's going on on your WIFI/LAN network and enable presence d ## ๐Ÿš€ Quick Start > [!WARNING] -> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://jokob-sk.github.io/NetAlertX/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. +> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://docs.netalertx.com/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. Start NetAlertX in seconds with Docker: @@ -60,14 +60,14 @@ docker compose up --force-recreate --build # To customize: edit docker-compose.yaml and run that last command again ``` -Need help configuring it? Check the [usage guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/README.md) or [full documentation](https://jokob-sk.github.io/NetAlertX/). +Need help configuring it? Check the [usage guide](https://docs.netalertx.com/README) or [full documentation](https://docs.netalertx.com/). For Home Assistant users: [Click here to add NetAlertX](https://my.home-assistant.io/redirect/supervisor_add_addon_repository/?repository_url=https%3A%2F%2Fgithub.com%2Falexbelgium%2Fhassio-addons) For other install methods, check the [installation docs](#-documentation) -| [๐Ÿ“‘ Docker guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_INSTALLATION.md) | [๐Ÿš€ Releases](https://github.com/jokob-sk/NetAlertX/releases) | [๐Ÿ“š Docs](https://jokob-sk.github.io/NetAlertX/) | [๐Ÿ”Œ Plugins](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md) | [๐Ÿค– Ask AI](https://gurubase.io/g/netalertx) +| [๐Ÿ“‘ Docker guide](https://docs.netalertx.com/DOCKER_INSTALLATION) | [๐Ÿš€ Releases](https://github.com/jokob-sk/NetAlertX/releases) | [๐Ÿ“š Docs](https://docs.netalertx.com/) | [๐Ÿ”Œ Plugins](https://docs.netalertx.com/PLUGINS) | [๐Ÿค– Ask AI](https://gurubase.io/g/netalertx) |----------------------| ----------------------| ----------------------| ----------------------| ----------------------| ![showcase][showcase] @@ -88,7 +88,7 @@ For other install methods, check the [installation docs](#-documentation) ### Scanners -The app scans your network for **New devices**, **New connections** (re-connections), **Disconnections**, **"Always Connected" devices down**, Devices **IP changes** and **Internet IP address changes**. Discovery & scan methods include: **arp-scan**, **Pi-hole - DB import**, **Pi-hole - DHCP leases import**, **Generic DHCP leases import**, **UNIFI controller import**, **SNMP-enabled router import**. Check the [Plugins](https://github.com/jokob-sk/NetAlertX/tree/main/docs/PLUGINS.md#readme) docs for a full list of avaliable plugins. +The app scans your network for **New devices**, **New connections** (re-connections), **Disconnections**, **"Always Connected" devices down**, Devices **IP changes** and **Internet IP address changes**. Discovery & scan methods include: **arp-scan**, **Pi-hole - DB import**, **Pi-hole - DHCP leases import**, **Generic DHCP leases import**, **UNIFI controller import**, **SNMP-enabled router import**. Check the [Plugins](https://docs.netalertx.com/PLUGINS#readme) docs for a full list of avaliable plugins. ### Notification gateways @@ -96,12 +96,12 @@ Send notifications to more than 80+ services, including Telegram via [Apprise](h ### Integrations and Plugins -Feed your data and device changes into [Home Assistant](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HOME_ASSISTANT.md), read [API endpoints](https://github.com/jokob-sk/NetAlertX/blob/main/docs/API.md), or use [Webhooks](https://github.com/jokob-sk/NetAlertX/blob/main/docs/WEBHOOK_N8N.md) to setup custom automation flows. You can also -build your own scanners with the [Plugin system](https://github.com/jokob-sk/NetAlertX/tree/main/docs/PLUGINS.md#readme) in as little as [15 minutes](https://www.youtube.com/watch?v=cdbxlwiWhv8). +Feed your data and device changes into [Home Assistant](https://docs.netalertx.com/HOME_ASSISTANT), read [API endpoints](https://docs.netalertx.com/API), or use [Webhooks](https://docs.netalertx.com/WEBHOOK_N8N) to setup custom automation flows. You can also +build your own scanners with the [Plugin system](https://docs.netalertx.com/PLUGINS#readme) in as little as [15 minutes](https://www.youtube.com/watch?v=cdbxlwiWhv8). ### Workflows -The [workflows module](https://github.com/jokob-sk/NetAlertX/blob/main/docs/WORKFLOWS.md) allows to automate repetitive tasks, making network management more efficient. Whether you need to assign newly discovered devices to a specific Network Node, auto-group devices from a given vendor, unarchive a device if detected online, or automatically delete devices, this module provides the flexibility to tailor the automations to your needs. +The [workflows module](https://docs.netalertx.com/WORKFLOWS) allows to automate repetitive tasks, making network management more efficient. Whether you need to assign newly discovered devices to a specific Network Node, auto-group devices from a given vendor, unarchive a device if detected online, or automatically delete devices, this module provides the flexibility to tailor the automations to your needs. ## ๐Ÿ“š Documentation @@ -109,15 +109,15 @@ The [workflows module](https://github.com/jokob-sk/NetAlertX/blob/main/docs/WORK Supported browsers: Chrome, Firefox -- [[Installation] Docker](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_INSTALLATION.md) +- [[Installation] Docker](https://docs.netalertx.com/DOCKER_INSTALLATION) - [[Installation] Home Assistant](https://github.com/alexbelgium/hassio-addons/tree/master/netalertx) -- [[Installation] Bare metal](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HW_INSTALL.md) +- [[Installation] Bare metal](https://docs.netalertx.com/HW_INSTALL) - [[Installation] Unraid App](https://unraid.net/community/apps) -- [[Setup] Usage and Configuration](https://github.com/jokob-sk/NetAlertX/blob/main/docs/README.md) -- [[Development] API docs](https://github.com/jokob-sk/NetAlertX/blob/main/docs/API.md) -- [[Development] Custom Plugins](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS_DEV.md) +- [[Setup] Usage and Configuration](https://docs.netalertx.com/README) +- [[Development] API docs](https://docs.netalertx.com/API) +- [[Development] Custom Plugins](https://docs.netalertx.com/PLUGINS_DEV) -...or explore all the [documentation here](https://jokob-sk.github.io/NetAlertX/). +...or explore all the [documentation here](https://docs.netalertx.com/). ## ๐Ÿ” Security & Privacy @@ -143,7 +143,7 @@ A: Yes, but some scanners (e.g. ARP) work best on Ethernet. For Wi-Fi, try SNMP, A: No. All scans and data remain local, unless you set up cloud-based notifications. **Q: Can I use this without Docker?** -A: Yes! You can install it bare-metal. See the [bare metal installation guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HW_INSTALL.md). +A: Yes! You can install it bare-metal. See the [bare metal installation guide](https://docs.netalertx.com/HW_INSTALL). **Q: Where is the data stored?** A: In the `/data/config` and `/data/db` folders. Back up these folders regularly. @@ -151,12 +151,12 @@ A: In the `/data/config` and `/data/db` folders. Back up these folders regularly ## ๐Ÿž Known Issues -- Some scanners (e.g. ARP) may not detect devices on different subnets. See the [Remote networks guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/REMOTE_NETWORKS.md) for workarounds. +- Some scanners (e.g. ARP) may not detect devices on different subnets. See the [Remote networks guide](https://docs.netalertx.com/REMOTE_NETWORKS) for workarounds. - Wi-Fi-only networks may require alternate scanners for accurate detection. - Notification throttling may be needed for large networks to prevent spam. - On some systems, elevated permissions (like `CAP_NET_RAW`) may be needed for low-level scanning. -Check the [GitHub Issues](https://github.com/jokob-sk/NetAlertX/issues) for the latest bug reports and solutions and consult [the official documentation](https://jokob-sk.github.io/NetAlertX/). +Check the [GitHub Issues](https://github.com/jokob-sk/NetAlertX/issues) for the latest bug reports and solutions and consult [the official documentation](https://docs.netalertx.com/). ## ๐Ÿ“ƒ Everything else diff --git a/back/app.conf b/back/app.conf index 469c4e8e..d0281eaa 100755 --- a/back/app.conf +++ b/back/app.conf @@ -33,7 +33,7 @@ NSLOOKUP_RUN='before_name_updates' AVAHISCAN_RUN='before_name_updates' NBTSCAN_RUN='before_name_updates' -# Email +# Email #------------------------------------- # (add SMTP to LOADED_PLUGINS to load) #------------------------------------- @@ -48,20 +48,19 @@ SMTP_PASS='password' SMTP_SKIP_TLS=False -# Webhook +# Webhook #------------------------------------- # (add WEBHOOK to LOADED_PLUGINS to load) #------------------------------------- WEBHOOK_RUN='disabled' # use 'on_notification' to enable WEBHOOK_URL='http://n8n.local:5555/webhook-test/aaaaaaaa-aaaa-aaaa-aaaaa-aaaaaaaaaaaa' -WEBHOOK_PAYLOAD='json' # webhook payload data format for the "body > attachements > text" attribute - # in https://github.com/jokob-sk/NetAlertX/blob/main/docs/webhook_json_sample.json +WEBHOOK_PAYLOAD='json' # webhook payload data format for the "body > attachements > text" attribute # supported values: 'json', 'html' or 'text' # e.g.: for discord use 'html' WEBHOOK_REQUEST_METHOD='GET' -# Apprise +# Apprise #------------------------------------- # (add APPRISE to LOADED_PLUGINS to load) #------------------------------------- @@ -71,7 +70,7 @@ APPRISE_URL='mailto://smtp-relay.sendinblue.com:587?from=user@gmail.com&name=app # NTFY -#------------------------------------- +#------------------------------------- # (add NTFY to LOADED_PLUGINS to load) #------------------------------------- NTFY_RUN='disabled' # use 'on_notification' to enable @@ -81,7 +80,7 @@ NTFY_USER='user' NTFY_PASSWORD='passw0rd' -# PUSHSAFER +# PUSHSAFER #------------------------------------- # (add PUSHSAFER to LOADED_PLUGINS to load) #------------------------------------- @@ -89,7 +88,7 @@ PUSHSAFER_RUN='disabled' # use 'on_notification' to enable PUSHSAFER_TOKEN='ApiKey' -# MQTT +# MQTT #------------------------------------- # (add MQTT to LOADED_PLUGINS to load) #------------------------------------- diff --git a/docs/API_LOGS.md b/docs/API_LOGS.md index 8907069d..81f85503 100644 --- a/docs/API_LOGS.md +++ b/docs/API_LOGS.md @@ -18,7 +18,6 @@ Only specific, pre-approved log files can be purged for security and stability r ``` app.log -app_front.log IP_changes.log stdout.log stderr.log diff --git a/docs/API_OLD.md b/docs/API_OLD.md index 2575f261..139cc701 100755 --- a/docs/API_OLD.md +++ b/docs/API_OLD.md @@ -1,7 +1,7 @@ # [Deprecated] API endpoints -> [!WARNING] -> Some of these endpoints will be deprecated soon. Please refere to the new [API](API.md) endpoints docs for details on the new API layer. +> [!WARNING] +> Some of these endpoints will be deprecated soon. Please refere to the new [API](API.md) endpoints docs for details on the new API layer. NetAlertX comes with a couple of API endpoints. All requests need to be authorized (executed in a logged in browser session) or you have to pass the value of the `API_TOKEN` settings as authorization bearer, for example: @@ -56,7 +56,7 @@ See also: [Debugging GraphQL issues](./DEBUG_API_SERVER.md) ### `curl` Command -You can use the following `curl` command to execute the query. +You can use the following `curl` command to execute the query. ```sh curl 'http://host:GRAPHQL_PORT/graphql' -X POST -H 'Authorization: Bearer API_TOKEN' -H 'Content-Type: application/json' --data '{ @@ -127,9 +127,9 @@ The response will be in JSON format, similar to the following: } ``` -## API Endpoint: JSON files +## API Endpoint: JSON files -This API endpoint retrieves static files, that are periodically updated. +This API endpoint retrieves static files, that are periodically updated. - Endpoint URL: `php/server/query_json.php?file=` - Host: `same as front end (web ui)` @@ -147,18 +147,18 @@ In the container, these files are located under the API directory (default: `/tm You can access the following files: - | File name | Description | - |----------------------|----------------------| + | File name | Description | + |----------------------|----------------------| | `notification_json_final.json` | The json version of the last notification (e.g. used for webhooks - [sample JSON](https://github.com/jokob-sk/NetAlertX/blob/main/front/report_templates/webhook_json_sample.json)). | - | `table_devices.json` | All of the available Devices detected by the app. | + | `table_devices.json` | All of the available Devices detected by the app. | | `table_plugins_events.json` | The list of the unprocessed (pending) notification events (plugins_events DB table). | - | `table_plugins_history.json` | The list of notification events history. | - | `table_plugins_objects.json` | The content of the plugins_objects table. Find more info on the [Plugin system here](https://github.com/jokob-sk/NetAlertX/tree/main/docs/PLUGINS.md)| - | `language_strings.json` | The content of the language_strings table, which in turn is loaded from the plugins `config.json` definitions. | + | `table_plugins_history.json` | The list of notification events history. | + | `table_plugins_objects.json` | The content of the plugins_objects table. Find more info on the [Plugin system here](https://docs.netalertx.com/PLUGINS)| + | `language_strings.json` | The content of the language_strings table, which in turn is loaded from the plugins `config.json` definitions. | | `table_custom_endpoint.json` | A custom endpoint generated by the SQL query specified by the `API_CUSTOM_SQL` setting. | | `table_settings.json` | The content of the settings table. | | `app_state.json` | Contains the current application state. | - + ### JSON Data format @@ -169,11 +169,11 @@ The endpoints starting with the `table_` prefix contain most, if not all, data c "data": [ { "db_column_name": "data", - "db_column_name2": "data2" - }, + "db_column_name2": "data2" + }, { "db_column_name": "data3", - "db_column_name2": "data4" + "db_column_name2": "data4" } ] } @@ -201,7 +201,7 @@ Example JSON of the `table_devices.json` endpoint with two Devices (database row "devParentMAC": "", "devParentPort": "", "devIcon": "globe" - }, + }, { "devMac": "a4:8f:ff:aa:ba:1f", "devName": "Net - USG", @@ -332,7 +332,7 @@ Grafana template sample: [Download json](./samples/API/Grafana_Dashboard.json) ## API Endpoint: /log files -This API endpoint retrieves files from the `/tmp/log` folder. +This API endpoint retrieves files from the `/tmp/log` folder. - Endpoint URL: `php/server/query_logs.php?file=` - Host: `same as front end (web ui)` @@ -357,7 +357,7 @@ This API endpoint retrieves files from the `/tmp/log` folder. ## API Endpoint: /config files -To retrieve files from the `/data/config` folder. +To retrieve files from the `/data/config` folder. - Endpoint URL: `php/server/query_config.php?file=` - Host: `same as front end (web ui)` diff --git a/docs/API_SESSIONS.md b/docs/API_SESSIONS.md index e5a4e746..94224aa4 100755 --- a/docs/API_SESSIONS.md +++ b/docs/API_SESSIONS.md @@ -118,11 +118,14 @@ curl -X DELETE "http://:/sessions/delete" \ ``` #### `curl` Example +**get sessions for mac** + ```bash curl -X GET "http://:/sessions/list?mac=AA:BB:CC:DD:EE:FF&start_date=2025-08-01&end_date=2025-08-21" \ -H "Authorization: Bearer " \ -H "Accept: application/json" ``` + --- ### Calendar View of Sessions diff --git a/docs/API_SSE.md b/docs/API_SSE.md new file mode 100644 index 00000000..f8e4f883 --- /dev/null +++ b/docs/API_SSE.md @@ -0,0 +1,78 @@ +# SSE (Server-Sent Events) + +Real-time app state updates via Server-Sent Events. Reduces server load ~95% vs polling. + +## Endpoints + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/sse/state` | GET | Stream state updates (requires Bearer token) | +| `/sse/stats` | GET | Debug: connected clients, queued events | + +## Usage + +### Connect to SSE Stream +```bash +curl -H "Authorization: Bearer YOUR_API_TOKEN" \ + http://localhost:5000/sse/state +``` + +### Check Connection Stats +```bash +curl -H "Authorization: Bearer YOUR_API_TOKEN" \ + http://localhost:5000/sse/stats +``` + +## Event Types + +- `state_update` - App state changed (e.g., "Scanning", "Processing") +- `unread_notifications_count_update` - Number of unread notifications changed (count: int) + +## Backend Integration + +Broadcasts automatically triggered in `app_state.py` via `broadcast_state_update()`: + +```python +from api_server.sse_broadcast import broadcast_state_update + +# Called on every state change - no additional code needed +broadcast_state_update(current_state="Scanning", settings_imported=time.time()) +``` + +## Frontend Integration + +Auto-enabled via `sse_manager.js`: + +```javascript +// In browser console: +netAlertXStateManager.getStats().then(stats => { + console.log("Connected clients:", stats.connected_clients); +}); +``` + +## Fallback Behavior + +- If SSE fails after 3 attempts, automatically switches to polling +- Polling starts at 1s, backs off to 30s max +- No user-visible difference in functionality + +## Files + +| File | Purpose | +|------|---------| +| `server/api_server/sse_endpoint.py` | SSE endpoints & event queue | +| `server/api_server/sse_broadcast.py` | Broadcast helper functions | +| `front/js/sse_manager.js` | Client-side SSE connection manager | + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| Connection refused | Check backend running, API token correct | +| No events received | Verify `broadcast_state_update()` is called on state changes | +| High memory | Events not processed fast enough, check client logs | +| Using polling instead of SSE | Normal fallback - check browser console for errors | + +--- + + diff --git a/docs/AUTHELIA.md b/docs/AUTHELIA.md index f0657716..1c181c52 100755 --- a/docs/AUTHELIA.md +++ b/docs/AUTHELIA.md @@ -1,8 +1,8 @@ ## Authelia support -> [!WARNING] -> -> This is community contributed content and work in progress. Contributions are welcome. +> [!NOTE] +> This is community-contributed. Due to environment, setup, or networking differences, results may vary. Please open a PR to improve it instead of creating an issue, as the maintainer is not actively maintaining it. + ```yaml theme: dark @@ -274,4 +274,4 @@ notifier: subject: "[Authelia] {title}" startup_check_address: postmaster@MYOTHERDOMAIN.LTD -``` \ No newline at end of file +``` diff --git a/docs/CNAME b/docs/CNAME new file mode 100644 index 00000000..260a7a30 --- /dev/null +++ b/docs/CNAME @@ -0,0 +1 @@ +docs.netalertx.com \ No newline at end of file diff --git a/docs/COMMUNITY_GUIDES.md b/docs/COMMUNITY_GUIDES.md index c8243118..f943aceb 100755 --- a/docs/COMMUNITY_GUIDES.md +++ b/docs/COMMUNITY_GUIDES.md @@ -1,15 +1,21 @@ # Community Guides -Use the official installation guides at first and use community content as supplementary material. Open an issue or PR if you'd like to add your link to the list ๐Ÿ™ (Ordered by last update time) +> [!NOTE] +> This is community-contributed. Due to environment, setup, or networking differences, results may vary. Please open a PR to improve it instead of creating an issue, as the maintainer is not actively maintaining it. + + +Use the official installation guides at first and use community content as supplementary material. (Ordered by last update time) - โ–ถ [Discover & Monitor Your Network with This Self-Hosted Open Source Tool - Lawrence Systems](https://www.youtube.com/watch?v=R3b5cxLZMpo) (June 2025) - โ–ถ [Home Lab Network Monitoring - Scotti-BYTE Enterprise Consulting Services](https://www.youtube.com/watch?v=0DryhzrQSJA) (July 2024) - ๐Ÿ“„ [How to Install NetAlertX on Your Synology NAS - Marius hosting](https://mariushosting.com/how-to-install-pi-alert-on-your-synology-nas/) (Updated frequently) - ๐Ÿ“„ [Using the PiAlert Network Security Scanner on a Raspberry Pi - PiMyLifeUp](https://pimylifeup.com/raspberry-pi-pialert/) -- โ–ถ [How to Setup Pi.Alert on Your Synology NAS - Digital Aloha](https://www.youtube.com/watch?v=M4YhpuRFaUg) +- โ–ถ [How to Setup Pi.Alert on Your Synology NAS - Digital Aloha](https://www.youtube.com/watch?v=M4YhpuRFaUg) - ๐Ÿ“„ [้˜ฒ่นญ็ฝ‘็ฅžๅ™จ๏ผŒ็ฝ‘็ปœๅฎ‰ๅ…จๅŠฉๆ‰‹ | ๆž็ฉบ้—ด้ƒจ็ฝฒ็ฝ‘็ปœๆ‰ซๆๅ’Œ้€š็Ÿฅ็ณป็ปŸใ€ŽNetAlertXใ€](https://blog.csdn.net/qq_63499861/article/details/141105273) - ๐Ÿ“„ [์‹œ๋†€/ํ—ค๋†€์—์„œ ๋„คํŠธ์›Œํฌ ์Šค์บ๋„ˆ Pi.Alert Docker๋กœ ์„ค์น˜ ๋ฐ ์‚ฌ์šฉํ•˜๊ธฐ](https://blog.dalso.org/article/%EC%8B%9C%EB%86%80-%ED%97%A4%EB%86%80%EC%97%90%EC%84%9C-%EB%84%A4%ED%8A%B8%EC%9B%8C%ED%81%AC-%EC%8A%A4%EC%BA%90%EB%84%88-pi-alert-docker%EB%A1%9C-%EC%84%A4%EC%B9%98-%EB%B0%8F-%EC%82%AC%EC%9A%A9) (July 2023) - ๐Ÿ“„ [็ฝ‘็ปœๅ…ฅไพตๆŽขๆต‹ๅ™จPi.Alert (Chinese)](https://codeantenna.com/a/VgUvIAjZ7J) (May 2023) - โ–ถ [Pi.Alert auf Synology & Docker by - Jรผrgen Barth](https://www.youtube.com/watch?v=-ouvA2UNu-A) (March 2023) - โ–ถ [Top Docker Container for Home Server Security - VirtualizationHowto](https://www.youtube.com/watch?v=tY-w-enLF6Q) (March 2023) - โ–ถ [Pi.Alert or WatchYourLAN can alert you to unknown devices appearing on your WiFi or LAN network - Danie van der Merwe](https://www.youtube.com/watch?v=v6an9QG2xF0) (November 2022) + + diff --git a/docs/CUSTOM_PROPERTIES.md b/docs/CUSTOM_PROPERTIES.md index 1961eb7b..669fe79a 100755 --- a/docs/CUSTOM_PROPERTIES.md +++ b/docs/CUSTOM_PROPERTIES.md @@ -13,31 +13,6 @@ This functionality allows you to define **custom properties** for devices, which --- -## Defining Custom Properties - -Custom properties are structured as a list of objects, where each property includes the following fields: - -| Field | Description | -|--------------------|-----------------------------------------------------------------------------| -| `CUSTPROP_icon` | The icon (Base64-encoded HTML) displayed for the property. | -| `CUSTPROP_type` | The action type (e.g., `show_notes`, `link`, `delete_dev`). | -| `CUSTPROP_name` | A short name or title for the property. | -| `CUSTPROP_args` | Arguments for the action (e.g., URL or modal text). | -| `CUSTPROP_notes` | Additional notes or details displayed when applicable. | -| `CUSTPROP_show` | A boolean to control visibility (`true` to show on the listing page). | - ---- - -## Available Action Types - -- **Show Notes**: Displays a modal with a title and additional notes. - - **Example**: Show firmware details or custom messages. -- **Link**: Redirects to a specified URL in the current browser tab. (**Arguments** Needs to contain the full URL.) -- **Link (New Tab)**: Opens a specified URL in a new browser tab. (**Arguments** Needs to contain the full URL.) -- **Delete Device**: Deletes the device using its MAC address. -- **Run Plugin**: Placeholder for executing custom plugins (not implemented yet). - ---- ## Usage on the Device Listing Page @@ -74,12 +49,39 @@ Visible properties (`CUSTPROP_show: true`) are displayed as interactive icons in 3. **Device Removal**: - Enable device removal functionality using `CUSTPROP_type: delete_dev`. +--- + +## Defining Custom Properties + +Custom properties are structured as a list of objects, where each property includes the following fields: + +| Field | Description | +|--------------------|-----------------------------------------------------------------------------| +| `CUSTPROP_icon` | The icon (Base64-encoded HTML) displayed for the property. | +| `CUSTPROP_type` | The action type (e.g., `show_notes`, `link`, `delete_dev`). | +| `CUSTPROP_name` | A short name or title for the property. | +| `CUSTPROP_args` | Arguments for the action (e.g., URL or modal text). | +| `CUSTPROP_notes` | Additional notes or details displayed when applicable. | +| `CUSTPROP_show` | A boolean to control visibility (`true` to show on the listing page). | + +--- + +## Available Action Types + +- **Show Notes**: Displays a modal with a title and additional notes. + - **Example**: Show firmware details or custom messages. +- **Link**: Redirects to a specified URL in the current browser tab. (**Arguments** Needs to contain the full URL.) +- **Link (New Tab)**: Opens a specified URL in a new browser tab. (**Arguments** Needs to contain the full URL.) +- **Delete Device**: Deletes the device using its MAC address. +- **Run Plugin**: Placeholder for executing custom plugins (not implemented yet). + + --- ## Notes - **Plugin Functionality**: The `run_plugin` action type is currently not implemented and will show an alert if used. -- **Custom Icons (Experimental ๐Ÿงช)**: Use Base64-encoded HTML to provide custom icons for each property. You can add your icons in Setttings via the `CUSTPROP_icon` settings +- **Custom Icons (Experimental ๐Ÿงช)**: Use Base64-encoded HTML to provide custom icons for each property. You can add your icons in Setttings via the `CUSTPROP_icon` settings - **Visibility Control**: Only properties with `CUSTPROP_show: true` will appear on the listing page. This feature provides a flexible way to enhance device management and display with interactive elements tailored to your needs. diff --git a/docs/DEVICES_BULK_EDITING.md b/docs/DEVICES_BULK_EDITING.md index 0e14081d..a0893d13 100755 --- a/docs/DEVICES_BULK_EDITING.md +++ b/docs/DEVICES_BULK_EDITING.md @@ -26,7 +26,7 @@ The database and device structure may change with new releases. When using the C ![Maintenance > CSV Export](./img/DEVICES_BULK_EDITING/MAINTENANCE_CSV_EXPORT.png) > [!NOTE] -> The file containing a list of Devices including the Network relationships between Network Nodes and connected devices. You can also trigger this by acessing this URL: `:20211/php/server/devices.php?action=ExportCSV` or via the `CSV Backup` plugin. (๐Ÿ’ก You can schedule this) +> The file containing a list of Devices including the Network relationships between Network Nodes and connected devices. You can also trigger this with the `CSV Backup` plugin. (๐Ÿ’ก You can schedule this) ![Settings > CSV Backup](./img/DEVICES_BULK_EDITING/CSV_BACKUP_SETTINGS.png) diff --git a/docs/DEVICE_MANAGEMENT.md b/docs/DEVICE_MANAGEMENT.md index f106da24..efdff7d7 100755 --- a/docs/DEVICE_MANAGEMENT.md +++ b/docs/DEVICE_MANAGEMENT.md @@ -13,7 +13,7 @@ The Main Info section is where most of the device identifiable information is st - **MAC**: MAC addres of the device. Not editable, unless creating a new dummy device. - **Last IP**: IP addres of the device. Not editable, unless creating a new dummy device. - - **Name**: Friendly device name. Autodetected via various ๐Ÿ†Ž Name discovery [plugins](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md). The app attaches `(IP match)` if the name is discovered via an IP match and not MAC match which could mean the name could be incorrect as IPs might change. + - **Name**: Friendly device name. Autodetected via various ๐Ÿ†Ž Name discovery [plugins](https://docs.netalertx.com/PLUGINS). The app attaches `(IP match)` if the name is discovered via an IP match and not MAC match which could mean the name could be incorrect as IPs might change. - **Icon**: Partially autodetected. Select an existing or [add a custom icon](./ICONS.md). You can also auto-apply the same icon on all devices of the same type. - **Owner**: Device owner (The list is self-populated with existing owners and you can add custom values). - **Type**: Select a device type from the dropdown list (`Smartphone`, `Tablet`, diff --git a/docs/DOCKER_COMPOSE.md b/docs/DOCKER_COMPOSE.md index 375cf5ad..396bc912 100755 --- a/docs/DOCKER_COMPOSE.md +++ b/docs/DOCKER_COMPOSE.md @@ -1,7 +1,7 @@ # NetAlertX and Docker Compose > [!WARNING] -> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://jokob-sk.github.io/NetAlertX/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. +> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://docs.netalertx.com/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. Great care is taken to ensure NetAlertX meets the needs of everyone while being flexible enough for anyone. This document outlines how you can configure your docker-compose. There are many settings, so we recommend using the Baseline Docker Compose as-is, or modifying it for your system.Good care is taken to ensure NetAlertX meets the needs of everyone while being flexible enough for anyone. This document outlines how you can configure your docker-compose. There are many settings, so we recommend using the Baseline Docker Compose as-is, or modifying it for your system. @@ -69,6 +69,8 @@ services: PORT: ${PORT:-20211} # Application port GRAPHQL_PORT: ${GRAPHQL_PORT:-20212} # GraphQL API port (passed into APP_CONF_OVERRIDE at runtime) # NETALERTX_DEBUG: ${NETALERTX_DEBUG:-0} # 0=kill all services and restart if any dies. 1 keeps running dead services. + # PUID: 20211 # Runtime PUID override, set to 0 to run as root + # PGID: 20211 # Runtime PGID override # Resource limits to prevent resource exhaustion mem_limit: 2048m # Maximum memory usage @@ -171,10 +173,6 @@ Now, any files created by NetAlertX in `/data/config` will appear in your `/loca This same method works for mounting other things, like custom plugins or enterprise NGINX files, as shown in the commented-out examples in the baseline file. -## Example Configuration Summaries - -Here are the essential modifications for common alternative setups. - ### Example 2: External `.env` File for Paths This method is useful for keeping your paths and other settings separate from your main compose file, making it more portable. diff --git a/docs/DOCKER_INSTALLATION.md b/docs/DOCKER_INSTALLATION.md index ad260362..2116fb33 100644 --- a/docs/DOCKER_INSTALLATION.md +++ b/docs/DOCKER_INSTALLATION.md @@ -6,7 +6,7 @@ # NetAlertX - Network scanner & notification framework -| [๐Ÿ“‘ Docker guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_INSTALLATION.md) | [๐Ÿš€ Releases](https://github.com/jokob-sk/NetAlertX/releases) | [๐Ÿ“š Docs](https://jokob-sk.github.io/NetAlertX/) | [๐Ÿ”Œ Plugins](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md) | [๐Ÿค– Ask AI](https://gurubase.io/g/netalertx) +| [๐Ÿ“‘ Docker guide](https://docs.netalertx.com/DOCKER_INSTALLATION) | [๐Ÿš€ Releases](https://github.com/jokob-sk/NetAlertX/releases) | [๐Ÿ“š Docs](https://docs.netalertx.com/) | [๐Ÿ”Œ Plugins](https://docs.netalertx.com/PLUGINS) | [๐Ÿค– Ask AI](https://gurubase.io/g/netalertx) |----------------------| ----------------------| ----------------------| ----------------------| ----------------------| @@ -16,12 +16,12 @@ Head to [https://netalertx.com/](https://netalertx.com/) for more gifs and screenshots ๐Ÿ“ท. > [!NOTE] -> There is also an experimental ๐Ÿงช [bare-metal install](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HW_INSTALL.md) method available. +> There is also an experimental ๐Ÿงช [bare-metal install](https://docs.netalertx.com/HW_INSTALL) method available. ## ๐Ÿ“• Basic Usage > [!WARNING] -> You will have to run the container on the `host` network and specify `SCAN_SUBNETS` unless you use other [plugin scanners](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md). The initial scan can take a few minutes, so please wait 5-10 minutes for the initial discovery to finish. +> You will have to run the container on the `host` network and specify `SCAN_SUBNETS` unless you use other [plugin scanners](https://docs.netalertx.com/PLUGINS). The initial scan can take a few minutes, so please wait 5-10 minutes for the initial discovery to finish. ```bash docker run -d --rm --network=host \ @@ -35,7 +35,7 @@ docker run -d --rm --network=host \ > Runtime UID/GID: The image defaults to a service user `netalertx` (UID/GID 20211). A separate readonly lock owner also uses UID/GID 20211 for 004/005 immutability. You can override the runtime UID/GID at build (ARG) or run (`--user` / compose `user:`) but must align writable mounts (`/data`, `/tmp*`) and tmpfs `uid/gid` to that choice. -See alternative [docked-compose examples](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_COMPOSE.md). +See alternative [docked-compose examples](https://docs.netalertx.com/DOCKER_COMPOSE). ### Default ports @@ -48,11 +48,11 @@ See alternative [docked-compose examples](https://github.com/jokob-sk/NetAlertX/ | Variable | Description | Example/Default Value | | :------------- |:------------------------| -----:| -| `PUID` |Runtime UID override | `20211` | +| `PUID` |Runtime UID override, set to `0` to run as root. | `20211` | | `PGID` |Runtime GID override | `20211` | | `PORT` |Port of the web interface | `20211` | | `LISTEN_ADDR` |Set the specific IP Address for the listener address for the nginx webserver (web interface). This could be useful when using multiple subnets to hide the web interface from all untrusted networks. | `0.0.0.0` | -|`LOADED_PLUGINS` | Default [plugins](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md) to load. Plugins cannot be loaded with `APP_CONF_OVERRIDE`, you need to use this variable instead and then specify the plugins settings with `APP_CONF_OVERRIDE`. | `["PIHOLE","ASUSWRT"]` | +|`LOADED_PLUGINS` | Default [plugins](https://docs.netalertx.com/PLUGINS) to load. Plugins cannot be loaded with `APP_CONF_OVERRIDE`, you need to use this variable instead and then specify the plugins settings with `APP_CONF_OVERRIDE`. | `["PIHOLE","ASUSWRT"]` | |`APP_CONF_OVERRIDE` | JSON override for settings (except `LOADED_PLUGINS`). | `{"SCAN_SUBNETS":"['192.168.1.0/24 --interface=eth1']","GRAPHQL_PORT":"20212"}` | |`ALWAYS_FRESH_INSTALL` | โš  If `true` will delete the content of the `/db` & `/config` folders. For testing purposes. Can be coupled with [watchtower](https://github.com/containrrr/watchtower) to have an always freshly installed `netalertx`/`netalertx-dev` image. | `true` | @@ -61,16 +61,16 @@ See alternative [docked-compose examples](https://github.com/jokob-sk/NetAlertX/ ### Docker paths > [!NOTE] -> See also [Backup strategies](https://github.com/jokob-sk/NetAlertX/blob/main/docs/BACKUPS.md). +> See also [Backup strategies](https://docs.netalertx.com/BACKUPS). | Required | Path | Description | | :------------- | :------------- | :-------------| | โœ… | `:/data` | Folder which needs to contain a `/db` and `/config` sub-folders. | | โœ… | `/etc/localtime:/etc/localtime:ro` | Ensuring the timezone is the same as on the server. | | | `:/tmp/log` | Logs folder useful for debugging if you have issues setting up the container | -| | `:/tmp/api` | The [API endpoint](https://github.com/jokob-sk/NetAlertX/blob/main/docs/API.md) containing static (but regularly updated) json and other files. Path configurable via `NETALERTX_API` environment variable. | -| | `:/app/front/plugins//ignore_plugin` | Map a file `ignore_plugin` to ignore a plugin. Plugins can be soft-disabled via settings. More in the [Plugin docs](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PLUGINS.md). | -| | `:/etc/resolv.conf` | Use a custom `resolv.conf` file for [better name resolution](https://github.com/jokob-sk/NetAlertX/blob/main/docs/REVERSE_DNS.md). | +| | `:/tmp/api` | The [API endpoint](https://docs.netalertx.com/API) containing static (but regularly updated) json and other files. Path configurable via `NETALERTX_API` environment variable. | +| | `:/app/front/plugins//ignore_plugin` | Map a file `ignore_plugin` to ignore a plugin. Plugins can be soft-disabled via settings. More in the [Plugin docs](https://docs.netalertx.com/PLUGINS). | +| | `:/etc/resolv.conf` | Use a custom `resolv.conf` file for [better name resolution](https://docs.netalertx.com/REVERSE_DNS). | ### Folder structure @@ -100,23 +100,23 @@ sudo chmod -R a+rwx /local_data_dir #### Setting up scanners -You have to specify which network(s) should be scanned. This is done by entering subnets that are accessible from the host. If you use the default `ARPSCAN` plugin, you have to specify at least one valid subnet and interface in the `SCAN_SUBNETS` setting. See the documentation on [How to set up multiple SUBNETS, VLANs and what are limitations](https://github.com/jokob-sk/NetAlertX/blob/main/docs/SUBNETS.md) for troubleshooting and more advanced scenarios. +You have to specify which network(s) should be scanned. This is done by entering subnets that are accessible from the host. If you use the default `ARPSCAN` plugin, you have to specify at least one valid subnet and interface in the `SCAN_SUBNETS` setting. See the documentation on [How to set up multiple SUBNETS, VLANs and what are limitations](https://docs.netalertx.com/SUBNETS) for troubleshooting and more advanced scenarios. -If you are running PiHole you can synchronize devices directly. Check the [PiHole configuration guide](https://github.com/jokob-sk/NetAlertX/blob/main/docs/PIHOLE_GUIDE.md) for details. +If you are running PiHole you can synchronize devices directly. Check the [PiHole configuration guide](https://docs.netalertx.com/PIHOLE_GUIDE) for details. > [!NOTE] -> You can bulk-import devices via the [CSV import method](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEVICES_BULK_EDITING.md). +> You can bulk-import devices via the [CSV import method](https://docs.netalertx.com/DEVICES_BULK_EDITING). #### Community guides -You can read or watch several [community configuration guides](https://github.com/jokob-sk/NetAlertX/blob/main/docs/COMMUNITY_GUIDES.md) in Chinese, Korean, German, or French. +You can read or watch several [community configuration guides](https://docs.netalertx.com/COMMUNITY_GUIDES) in Chinese, Korean, German, or French. > Please note these might be outdated. Rely on official documentation first. #### Common issues - Before creating a new issue, please check if a similar issue was [already resolved](https://github.com/jokob-sk/NetAlertX/issues?q=is%3Aissue+is%3Aclosed). -- Check also common issues and [debugging tips](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEBUG_TIPS.md). +- Check also common issues and [debugging tips](https://docs.netalertx.com/DEBUG_TIPS). ## ๐Ÿ’™ Support me diff --git a/docs/DOCKER_MAINTENANCE.md b/docs/DOCKER_MAINTENANCE.md index 89e35afd..e38b3af6 100644 --- a/docs/DOCKER_MAINTENANCE.md +++ b/docs/DOCKER_MAINTENANCE.md @@ -1,7 +1,7 @@ # The NetAlertX Container Operator's Guide > [!WARNING] -> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://jokob-sk.github.io/NetAlertX/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. +> โš ๏ธ **Important:** The docker-compose has recently changed. Carefully read the [Migration guide](https://docs.netalertx.com/MIGRATION/?h=migrat#12-migration-from-netalertx-v25524) for detailed instructions. This guide assumes you are starting with the official `docker-compose.yml` file provided with the project. We strongly recommend you start with or migrate to this file as your baseline and modify it to suit your specific needs (e.g., changing file paths). While there are many ways to configure NetAlertX, the default file is designed to meet the mandatory security baseline with layer-2 networking capabilities while operating securely and without startup warnings. diff --git a/docs/DOCKER_SWARM.md b/docs/DOCKER_SWARM.md index f1af830c..3d0d218d 100755 --- a/docs/DOCKER_SWARM.md +++ b/docs/DOCKER_SWARM.md @@ -1,5 +1,9 @@ # Docker Swarm Deployment Guide (IPvlan) +> [!NOTE] +> This is community-contributed. Due to environment, setup, or networking differences, results may vary. Please open a PR to improve it instead of creating an issue, as the maintainer is not actively maintaining it. + + This guide describes how to deploy **NetAlertX** in a **Docker Swarm** environment using an `ipvlan` network. This enables the container to receive a LAN IP address directly, which is ideal for network monitoring. --- @@ -68,4 +72,3 @@ networks: * Make sure the assigned IP (`192.168.1.240` above) is not in use or managed by DHCP. * You may also use a node label constraint instead of `node.role == manager` for more control. - diff --git a/docs/FILE_PERMISSIONS.md b/docs/FILE_PERMISSIONS.md index 96082893..4621607e 100755 --- a/docs/FILE_PERMISSIONS.md +++ b/docs/FILE_PERMISSIONS.md @@ -38,16 +38,28 @@ NetAlertX requires certain paths to be writable at runtime. These paths should b > All these paths will have **UID 20211 / GID 20211** inside the container. Files on the host will appear owned by `20211:20211`. -You can cahnge the default PUID and GUID with env variables: +## Running as `root` + +You can override the default PUID and PGID using environment variables: ```yaml ... environment: - PUID: 20211 # Runtime PUID override + PUID: 20211 # Runtime PUID override, set to 0 to run as root PGID: 20211 # Runtime PGID override ... ``` +To run as the root user, it usually looks like this (verify the IDs on your server first by executing `id root`): + +```yaml +... + environment: + PUID: 0 # Runtime PUID override, set to 0 to run as root + PGID: 100 # Runtime PGID override +... +``` + ### Solution 1. **Run the container once as root** (`--user "0"`) to allow it to correct permissions automatically: diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index 9f8eb115..8e6cbccc 100755 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -2,24 +2,24 @@ ## Installation options -NetAlertX can be installed several ways. The best supported option is Docker, followed by a supervised Home Assistant instance, as an Unraid app, and lastly, on bare metal. +NetAlertX can be installed several ways. The best supported option is Docker, followed by a supervised Home Assistant instance, as an Unraid app, and lastly, on bare metal. -- [[Installation] Docker (recommended)](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_INSTALLATION.md) -- [[Installation] Home Assistant](https://github.com/alexbelgium/hassio-addons/tree/master/netalertx) -- [[Installation] Unraid App](https://unraid.net/community/apps) -- [[Installation] Bare metal (experimental - looking for maintainers)](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HW_INSTALL.md) +- [[Installation] Docker (recommended)](https://docs.netalertx.com/DOCKER_INSTALLATION) +- [[Installation] Home Assistant](https://github.com/alexbelgium/hassio-addons/tree/master/netalertx) +- [[Installation] Unraid App](https://unraid.net/community/apps) +- [[Installation] Bare metal (experimental - looking for maintainers)](https://docs.netalertx.com/HW_INSTALL) ## Help -If facing issues, please spend a few minutes seraching. +If facing issues, please spend a few minutes seraching. - Check [common issues](./COMMON_ISSUES.md) -- Have a look at [Community guides](./COMMUNITY_GUIDES.md) -- [Search closed or open issues or discussions](https://github.com/jokob-sk/NetAlertX/issues?q=is%3Aissue) +- Have a look at [Community guides](./COMMUNITY_GUIDES.md) +- [Search closed or open issues or discussions](https://github.com/jokob-sk/NetAlertX/issues?q=is%3Aissue) - Check [Discord](https://discord.gg/NczTUTWyRr) > [!NOTE] -> If you can't find a solution anywhere, ask in Discord if you think it's a quick question, otherwise open a new [issue](https://github.com/jokob-sk/NetAlertX/issues/new?template=setup-help.yml). Please fill in as much as possible to speed up the help process. +> If you can't find a solution anywhere, ask in Discord if you think it's a quick question, otherwise open a new [issue](https://github.com/jokob-sk/NetAlertX/issues/new?template=setup-help.yml). Please fill in as much as possible to speed up the help process. > diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md index 24aaad04..9e39e5ba 100755 --- a/docs/MIGRATION.md +++ b/docs/MIGRATION.md @@ -297,5 +297,5 @@ sudo chmod -R a+rwx /local_data_dir ``` 8. Start the container and verify everything works as expeexpected. -9. Check the [Permissions -> Writable-paths](https://jokob-sk.github.io/NetAlertX/FILE_PERMISSIONS/#writable-paths) what directories to mount if you'd like to access the API or log files. - +9. Check the [Permissions -> Writable-paths](https://docs.netalertx.com/FILE_PERMISSIONS/#writable-paths) what directories to mount if you'd like to access the API or log files. + diff --git a/docs/PLUGINS_DEV.md b/docs/PLUGINS_DEV.md index 1dd93ed1..3fbd8bff 100755 --- a/docs/PLUGINS_DEV.md +++ b/docs/PLUGINS_DEV.md @@ -1,22 +1,42 @@ -# Creating a custom plugin +# Plugin Development Guide -NetAlertX comes with a plugin system to feed events from third-party scripts into the UI and then send notifications, if desired. The highlighted core functionality this plugin system supports, is: - -* dynamic creation of a simple UI to interact with the discovered objects, -* filtering of displayed values in the Devices UI -* surface settings of plugins in the UI, -* different column types for reported values to e.g. link back to a device -* import objects into existing NetAlertX database tables - -> (Currently, update/overwriting of existing objects is only supported for devices via the `CurrentScan` table.) - -> [!NOTE] -> For a high-level overview of how the `config.json` is used and it's lifecycle check the [config.json Lifecycle in NetAlertX Guide](PLUGINS_DEV_CONFIG.md). - -### ๐ŸŽฅ Watch the video: +This comprehensive guide covers how to build plugins for NetAlertX. > [!TIP] -> Read this guide [Development environment setup guide](./DEV_ENV_SETUP.md) to set up your local environment for development. ๐Ÿ‘ฉโ€๐Ÿ’ป +> **New to plugin development?** Start with the [Quick Start Guide](PLUGINS_DEV_QUICK_START.md) to get a working plugin in 5 minutes. + +NetAlertX comes with a plugin system to feed events from third-party scripts into the UI and then send notifications, if desired. The highlighted core functionality this plugin system supports: + +* **Dynamic UI generation** - Automatically create tables for discovered objects +* **Data filtering** - Filter and link values in the Devices UI +* **User settings** - Surface plugin configuration in the Settings UI +* **Rich display types** - Color-coded badges, links, formatted text, and more +* **Database integration** - Import plugin data into NetAlertX tables like `CurrentScan` or `Devices` + +> [!NOTE] +> For a high-level overview of how the `config.json` is used and its lifecycle, see the [config.json Lifecycle Guide](PLUGINS_DEV_CONFIG.md). + +## Quick Links + +### ๐Ÿš€ Getting Started +- **[Quick Start Guide](PLUGINS_DEV_QUICK_START.md)** - Create a working plugin in 5 minutes +- **[Development Environment Setup](./DEV_ENV_SETUP.md)** - Set up your local development environment + +### ๐Ÿ“š Core Concepts +- **[Data Contract](PLUGINS_DEV_DATA_CONTRACT.md)** - The exact output format plugins must follow (9-13 columns, pipe-delimited) +- **[Data Sources](PLUGINS_DEV_DATASOURCES.md)** - How plugins retrieve data (scripts, databases, templates) +- **[Plugin Settings System](PLUGINS_DEV_SETTINGS.md)** - Let users configure your plugin via the UI +- **[UI Components](PLUGINS_DEV_UI_COMPONENTS.md)** - Display plugin results with color coding, links, and more + +### ๐Ÿ—๏ธ Architecture +- **[Plugin Config Lifecycle](PLUGINS_DEV_CONFIG.md)** - How `config.json` is loaded and used +- **[Full Plugin Development Reference](#full-reference-below)** - Comprehensive details on all aspects + +### ๐Ÿ› Troubleshooting +- **[Debugging Plugins](DEBUG_PLUGINS.md)** - Troubleshoot plugin issues +- **[Plugin Examples](../front/plugins)** - Study existing plugins as reference implementations + +### ๐ŸŽฅ Video Tutorial [![Watch the video](./img/YouTube_thumbnail.png)](https://youtu.be/cdbxlwiWhv8) @@ -26,770 +46,351 @@ NetAlertX comes with a plugin system to feed events from third-party scripts int |----------------------|----------------------| ----------------------| | ![Screen 4][screen4] | ![Screen 5][screen5] | -## Use cases +## Use Cases -Example use cases for plugins could be: +Plugins are infinitely flexible. Here are some examples: -* Monitor a web service and alert me if it's down -* Import devices from dhcp.leases files instead/complementary to using PiHole or arp-scans -* Creating ad-hoc UI tables from existing data in the NetAlertX database, e.g. to show all open ports on devices, to list devices that disconnected in the last hour, etc. -* Using other device discovery methods on the network and importing the data as new devices -* Creating a script to create FAKE devices based on user input via custom settings -* ...at this point the limitation is mostly the creativity rather than the capability (there might be edge cases and a need to support more form controls for user input off custom settings, but you probably get the idea) +* **Device Discovery** - Scan networks using ARP, mDNS, DHCP leases, or custom protocols +* **Service Monitoring** - Monitor web services, APIs, or network services for availability +* **Integration** - Import devices from PiHole, Home Assistant, Unifi, or other systems +* **Enrichment** - Add data like geolocation, threat intelligence, or asset metadata +* **Alerting** - Send notifications to Slack, Discord, Telegram, email, or webhooks +* **Reporting** - Generate insights from existing NetAlertX database (open ports, recent changes, etc.) +* **Custom Logic** - Create fake devices, trigger automations, or implement custom heuristics -If you wish to develop a plugin, please check the existing plugin structure. Once the settings are saved by the user they need to be removed from the `app.conf` file manually if you want to re-initialize them from the `config.json` of the plugin. +If you can imagine it and script it, you can build a plugin. -## โš  Disclaimer +## Limitations & Notes -Please read the below carefully if you'd like to contribute with a plugin yourself. This documentation file might be outdated, so double-check the existing plugins as well. +- Plugin data is deduplicated hourly (same Primary ID + Secondary ID + User Data = duplicate removed) +- Currently, only `CurrentScan` table supports update/overwrite of existing objects +- Plugin results must follow the strict [Data Contract](PLUGINS_DEV_DATA_CONTRACT.md) +- Plugins run with the same permissions as the NetAlertX process +- External dependencies must be installed in the container -## Plugin development quick start +## Plugin Development Workflow -1. Create a new folder for your plugin (e.g. `my_plugin`) -1. Copy the files from the `__template` folder into the newly created folder -1. Update the relevant attributes in the `config.json` file, especially `code_name` and `unique_prefix`, e.g.: - - `"code_name": "my_plugin"` - must match the folder name - - `"unique_prefix": "MYPLG"` - has to be unique, upper case letters only -1. Update the `RUN` setting to point to your script file -1. Update the rest of the `config.json` sections and implement the actual data retrieval in our python script +### Step 1: Understand the Basics +1. Read [Quick Start Guide](PLUGINS_DEV_QUICK_START.md) - 5 minute overview +2. Study the [Data Contract](PLUGINS_DEV_DATA_CONTRACT.md) - Understand the output format +3. Choose a [Data Source](PLUGINS_DEV_DATASOURCES.md) - Where does your data come from? -## Plugin file structure overview +### Step 2: Create Your Plugin +1. Copy the `__template` plugin folder (see below for structure) +2. Update `config.json` with your plugin metadata +3. Implement `script.py` (or configure alternative data source) +4. Test locally in the devcontainer -> โš ๏ธFolder name must be the same as the code name value in: `"code_name": ""` -> Unique prefix needs to be unique compared to the other settings prefixes, e.g.: the prefix `APPRISE` is already in use. +### Step 3: Configure & Display +1. Define [Settings](PLUGINS_DEV_SETTINGS.md) for user configuration +2. Design [UI Components](PLUGINS_DEV_UI_COMPONENTS.md) for result display +3. Map to database tables if needed (for notifications, etc.) - | File | Required (plugin type) | Description | - |----------------------|----------------------|----------------------| - | `config.json` | yes | Contains the plugin configuration (manifest) including the settings available to the user. | - | `script.py` | no | The Python script itself. You may call any valid linux command. | - | `last_result..log` | no | The file used to interface between NetAlertX and the plugin. Required for a script plugin if you want to feed data into the app. Stored in the `/api/log/plugins/` | - | `README.md` | yes | Any setup considerations or overview | +### Step 4: Deploy & Test +1. Restart the backend +2. Test via Settings โ†’ Plugin Settings +3. Verify results in UI and logs +4. Check `/tmp/log/plugins/last_result..log` + +See [Quick Start Guide](PLUGINS_DEV_QUICK_START.md) for detailed step-by-step instructions. + +## Plugin File Structure + +Every plugin lives in its own folder under `/app/front/plugins/`. + +> **Important:** Folder name must match the `"code_name"` value in `config.json` + +``` +/app/front/plugins/ +โ”œโ”€โ”€ __template/ # Copy this as a starting point +โ”‚ โ”œโ”€โ”€ config.json # Plugin manifest (configuration) +โ”‚ โ”œโ”€โ”€ script.py # Your plugin logic (optional, depends on data_source) +โ”‚ โ””โ”€โ”€ README.md # Setup and usage documentation +โ”œโ”€โ”€ my_plugin/ # Your new plugin +โ”‚ โ”œโ”€โ”€ config.json # REQUIRED - Plugin manifest +โ”‚ โ”œโ”€โ”€ script.py # OPTIONAL - Python script (if using script data source) +โ”‚ โ”œโ”€โ”€ README.md # REQUIRED - Documentation for users +โ”‚ โ””โ”€โ”€ other_files... # Your supporting files +``` + +## Plugin Manifest (config.json) + +The `config.json` file is the **plugin manifest** - it tells NetAlertX everything about your plugin: + +- **Metadata:** Plugin name, description, icon +- **Execution:** When to run, what command to run, timeout +- **Settings:** User-configurable options +- **Data contract:** Column definitions and how to display results +- **Integration:** Database mappings, notifications, filters + +**Example minimal config.json:** + +```json +{ + "code_name": "my_plugin", + "unique_prefix": "MYPLN", + "display_name": [{"language_code": "en_us", "string": "My Plugin"}], + "description": [{"language_code": "en_us", "string": "My awesome plugin"}], + "icon": "fa-plug", + "data_source": "script", + "execution_order": "Layer_0", + "settings": [ + { + "function": "RUN", + "type": {"dataType": "string", "elements": [{"elementType": "select", "elementOptions": [], "transformers": []}]}, + "default_value": "disabled", + "options": ["disabled", "once", "schedule"], + "localized": ["name"], + "name": [{"language_code": "en_us", "string": "When to run"}] + }, + { + "function": "CMD", + "type": {"dataType": "string", "elements": [{"elementType": "input", "elementOptions": [], "transformers": []}]}, + "default_value": "python3 /app/front/plugins/my_plugin/script.py", + "localized": ["name"], + "name": [{"language_code": "en_us", "string": "Command"}] + } + ], + "database_column_definitions": [] +} +``` + +> For comprehensive `config.json` documentation, see [PLUGINS_DEV_CONFIG.md](PLUGINS_DEV_CONFIG.md) + +## Full Reference (Below) + +The sections below provide complete reference documentation for all plugin development topics. Use the quick links above to jump to specific sections, or read sequentially for a deep dive. More on specifics below. -### Column order and values (plugins interface contract) +--- -> [!IMPORTANT] -> Spend some time reading and trying to understand the below table. This is the interface between the Plugins and the core application. The application expets 9 or 13 values The first 9 values are mandatory. The next 4 values (`HelpVal1` to `HelpVal4`) are optional. However, if you use any of these optional values (e.g., `HelpVal1`), you need to supply all optional values (e.g., `HelpVal2`, `HelpVal3`, and `HelpVal4`). If a value is not used, it should be padded with `null`. +## Data Contract & Output Format - | Order | Represented Column | Value Required | Description | - |----------------------|----------------------|----------------------|----------------------| - | 0 | `Object_PrimaryID` | yes | The primary ID used to group Events under. | - | 1 | `Object_SecondaryID` | no | Optional secondary ID to create a relationship beween other entities, such as a MAC address | - | 2 | `DateTime` | yes | When the event occured in the format `2023-01-02 15:56:30` | - | 3 | `Watched_Value1` | yes | A value that is watched and users can receive notifications if it changed compared to the previously saved entry. For example IP address | - | 4 | `Watched_Value2` | no | As above | - | 5 | `Watched_Value3` | no | As above | - | 6 | `Watched_Value4` | no | As above | - | 7 | `Extra` | no | Any other data you want to pass and display in NetAlertX and the notifications | - | 8 | `ForeignKey` | no | A foreign key that can be used to link to the parent object (usually a MAC address) | - | 9 | `HelpVal1` | no | (optional) A helper value | - | 10 | `HelpVal2` | no | (optional) A helper value | - | 11 | `HelpVal3` | no | (optional) A helper value | - | 12 | `HelpVal4` | no | (optional) A helper value | +For detailed information on plugin output format, see **[PLUGINS_DEV_DATA_CONTRACT.md](PLUGINS_DEV_DATA_CONTRACT.md)**. +Quick reference: +- **Format:** Pipe-delimited (`|`) text file +- **Location:** `/tmp/log/plugins/last_result..log` +- **Columns:** 9 required + 4 optional = 13 maximum +- **Helper:** Use `plugin_helper.py` for easy formatting -> [!NOTE] -> De-duplication is run once an hour on the `Plugins_Objects` database table and duplicate entries with the same value in columns `Object_PrimaryID`, `Object_SecondaryID`, `Plugin` (auto-filled based on `unique_prefix` of the plugin), `UserData` (can be populated with the `"type": "textbox_save"` column type) are removed. +### The 9 Mandatory Columns -# config.json structure +| Column | Name | Required | Example | +|--------|------|----------|---------| +| 0 | Object_PrimaryID | **YES** | `"device_name"` or `"192.168.1.1"` | +| 1 | Object_SecondaryID | no | `"secondary_id"` or `null` | +| 2 | DateTime | **YES** | `"2023-01-02 15:56:30"` | +| 3 | Watched_Value1 | **YES** | `"online"` or `"200"` | +| 4 | Watched_Value2 | no | `"ip_address"` or `null` | +| 5 | Watched_Value3 | no | `null` | +| 6 | Watched_Value4 | no | `null` | +| 7 | Extra | no | `"additional data"` or `null` | +| 8 | ForeignKey | no | `"aa:bb:cc:dd:ee:ff"` or `null` | -The `config.json` file is the manifest of the plugin. It contains mainly settings definitions and the mapping of Plugin objects to NetAlertX objects. +See [Data Contract](PLUGINS_DEV_DATA_CONTRACT.md) for examples, validation, and debugging tips. -## Execution order +--- -The execution order is used to specify when a plugin is executed. This is useful if a plugin has access and surfaces more information than others. If a device is detected by 2 plugins and inserted into the `CurrentScan` table, the plugin with the higher priority (e.g.: `Level_0` is a higher priority than `Level_1`) will insert it's values first. These values (devices) will be then prioritized over any values inserted later. +## Config.json: Settings & Configuration + +For detailed settings documentation, see **[PLUGINS_DEV_SETTINGS.md](PLUGINS_DEV_SETTINGS.md)** and **[PLUGINS_DEV_DATASOURCES.md](PLUGINS_DEV_DATASOURCES.md)**. + +### Setting Object Structure + +Every setting in your plugin has this structure: ```json { - "execution_order" : "Layer_0" + "function": "UNIQUE_CODE", + "type": {"dataType": "string", "elements": [...]}, + "default_value": "...", + "options": [...], + "localized": ["name", "description"], + "name": [{"language_code": "en_us", "string": "Display Name"}], + "description": [{"language_code": "en_us", "string": "Help text"}] } ``` -## Supported data sources +### Reserved Function Names -Currently, these data sources are supported (valid `data_source` value). +These control core plugin behavior: -| Name | `data_source` value | Needs to return a "table"* | Overview (more details on this page below) | -|----------------------|----------------------|----------------------|----------------------| -| Script | `script` | no | Executes any linux command in the `CMD` setting. | -| NetAlertX DB query | `app-db-query` | yes | Executes a SQL query on the NetAlertX database in the `CMD` setting. | -| Template | `template` | no | Used to generate internal settings, such as default values. | -| External SQLite DB query | `sqlite-db-query` | yes | Executes a SQL query from the `CMD` setting on an external SQLite database mapped in the `DB_PATH` setting. | -| Plugin type | `plugin_type` | no | Specifies the type of the plugin and in which section the Plugin settings are displayed ( one of `general/system/scanner/other/publisher` ). | +| Function | Purpose | Required | Options | +|----------|---------|----------|---------| +| `RUN` | When to execute | **YES** | `disabled`, `once`, `schedule`, `always_after_scan`, `before_name_updates`, `on_new_device` | +| `RUN_SCHD` | Cron schedule | If `RUN=schedule` | Cron format: `"0 * * * *"` | +| `CMD` | Command to run | **YES** | Shell command or script path | +| `RUN_TIMEOUT` | Max execution time | optional | Seconds: `"60"` | +| `WATCH` | Monitor for changes | optional | Column names | +| `REPORT_ON` | When to notify | optional | `new`, `watched-changed`, `watched-not-changed`, `missing-in-last-scan` | +| `DB_PATH` | External DB path | If using SQLite | `/path/to/db.db` | -> * "Needs to return a "table" means that the application expects a `last_result..log` file with some results. It's not a blocker, however warnings in the `app.log` might be logged. +See [PLUGINS_DEV_SETTINGS.md](PLUGINS_DEV_SETTINGS.md) for full component types and examples. -> ๐Ÿ”ŽExample ->```json ->"data_source": "app-db-query" ->``` -If you want to display plugin objects or import devices into the app, data sources have to return a "table" of the exact structure as outlined above. +--- -You can show or hide the UI on the "Plugins" page and "Plugins" tab for a plugin on devices via the `show_ui` property: +## Filters & Data Display -> ๐Ÿ”ŽExample ->```json -> "show_ui": true, -> ``` +For comprehensive display configuration, see **[PLUGINS_DEV_UI_COMPONENTS.md](PLUGINS_DEV_UI_COMPONENTS.md)**. -### "data_source": "script" +### Filters - If the `data_source` is set to `script` the `CMD` setting (that you specify in the `settings` array section in the `config.json`) contains an executable Linux command, that usually generates a `last_result..log` file (not required if you don't import any data into the app). The `last_result..log` file needs to be saved in `/api/log/plugins`. +Control which rows display in the UI: -> [!IMPORTANT] -> A lot of the work is taken care of by the [`plugin_helper.py` library](/front/plugins/plugin_helper.py). You don't need to manage the `last_result..log` file if using the helper objects. Check other `script.py` of other plugins for details. - - The content of the `last_result..log` file needs to contain the columns as defined in the "Column order and values" section above. The order of columns can't be changed. After every scan it should contain only the results from the latest scan/execution. - -- The format of the `last_result..log` is a `csv`-like file with the pipe `|` as a separator. -- 9 (nine) values need to be supplied, so every line needs to contain 8 pipe separators. Empty values are represented by `null`. -- Don't render "headers" for these "columns". -Every scan result/event entry needs to be on a new line. -- You can find which "columns" need to be present, and if the value is required or optional, in the "Column order and values" section. -- The order of these "columns" can't be changed. - -#### ๐Ÿ”Ž last_result.prefix.log examples - -Valid CSV: - -```csv - -https://www.google.com|null|2023-01-02 15:56:30|200|0.7898|null|null|null|null -https://www.duckduckgo.com|192.168.0.1|2023-01-02 15:56:30|200|0.9898|null|null|Best search engine|ff:ee:ff:11:ff:11 - -``` - -Invalid CSV with different errors on each line: - -```csv - -https://www.google.com|null|2023-01-02 15:56:30|200|0.7898||null|null|null -https://www.duckduckgo.com|null|2023-01-02 15:56:30|200|0.9898|null|null|Best search engine| -|https://www.duckduckgo.com|null|2023-01-02 15:56:30|200|0.9898|null|null|Best search engine|null -null|192.168.1.1|2023-01-02 15:56:30|200|0.9898|null|null|Best search engine -https://www.duckduckgo.com|192.168.1.1|2023-01-02 15:56:30|null|0.9898|null|null|Best search engine -https://www.google.com|null|2023-01-02 15:56:30|200|0.7898||| -https://www.google.com|null|2023-01-02 15:56:30|200|0.7898| - -``` - -### "data_source": "app-db-query" - -If the `data_source` is set to `app-db-query`, the `CMD` setting needs to contain a SQL query rendering the columns as defined in the "Column order and values" section above. The order of columns is important. - -This SQL query is executed on the `app.db` SQLite database file. - -> ๐Ÿ”ŽExample -> -> SQL query example: -> -> ```SQL -> SELECT dv.devName as Object_PrimaryID, -> cast(dv.devLastIP as VARCHAR(100)) || ':' || cast( SUBSTR(ns.Port ,0, INSTR(ns.Port , '/')) as VARCHAR(100)) as Object_SecondaryID, -> datetime() as DateTime, -> ns.Service as Watched_Value1, -> ns.State as Watched_Value2, -> 'null' as Watched_Value3, -> 'null' as Watched_Value4, -> ns.Extra as Extra, -> dv.devMac as ForeignKey -> FROM -> (SELECT * FROM Nmap_Scan) ns -> LEFT JOIN -> (SELECT devName, devMac, devLastIP FROM Devices) dv -> ON ns.MAC = dv.devMac -> ``` -> -> Required `CMD` setting example with above query (you can set `"type": "label"` if you want it to make uneditable in the UI): -> -> ```json -> { -> "function": "CMD", -> "type": {"dataType":"string", "elements": [{"elementType" : "input", "elementOptions" : [] ,"transformers": []}]}, -> "default_value":"SELECT dv.devName as Object_PrimaryID, cast(dv.devLastIP as VARCHAR(100)) || ':' || cast( SUBSTR(ns.Port ,0, INSTR(ns.Port , '/')) as VARCHAR(100)) as Object_SecondaryID, datetime() as DateTime, ns.Service as Watched_Value1, ns.State as Watched_Value2, 'null' as Watched_Value3, 'null' as Watched_Value4, ns.Extra as Extra FROM (SELECT * FROM Nmap_Scan) ns LEFT JOIN (SELECT devName, devMac, devLastIP FROM Devices) dv ON ns.MAC = dv.devMac", -> "options": [], -> "localized": ["name", "description"], -> "name" : [{ -> "language_code":"en_us", -> "string" : "SQL to run" -> }], -> "description": [{ -> "language_code":"en_us", -> "string" : "This SQL query is used to populate the coresponding UI tables under the Plugins section." -> }] -> } -> ``` - -### "data_source": "template" - -In most cases, it is used to initialize settings. Check the `newdev_template` plugin for details. - -### "data_source": "sqlite-db-query" - -You can execute a SQL query on an external database connected to the current NetAlertX database via a temporary `EXTERNAL_.` prefix. - -For example for `PIHOLE` (`"unique_prefix": "PIHOLE"`) it is `EXTERNAL_PIHOLE.`. The external SQLite database file has to be mapped in the container to the path specified in the `DB_PATH` setting: - -> ๐Ÿ”ŽExample -> ->```json -> ... ->{ -> "function": "DB_PATH", -> "type": {"dataType":"string", "elements": [{"elementType" : "input", "elementOptions" : [{"readonly": "true"}] ,"transformers": []}]}, -> "default_value":"/etc/pihole/pihole-FTL.db", -> "options": [], -> "localized": ["name", "description"], -> "name" : [{ -> "language_code":"en_us", -> "string" : "DB Path" -> }], -> "description": [{ -> "language_code":"en_us", -> "string" : "Required setting for the sqlite-db-query plugin type. Is used to mount an external SQLite database and execute the SQL query stored in the CMD setting." -> }] -> } -> ... ->``` - -The actual SQL query you want to execute is then stored as a `CMD` setting, similar to a Plugin of the `app-db-query` plugin type. The format has to adhere to the format outlined in the "Column order and values" section above. - -> ๐Ÿ”ŽExample -> -> Notice the `EXTERNAL_PIHOLE.` prefix. -> ->```json ->{ -> "function": "CMD", -> "type": {"dataType":"string", "elements": [{"elementType" : "input", "elementOptions" : [] ,"transformers": []}]}, -> "default_value":"SELECT hwaddr as Object_PrimaryID, cast('http://' || (SELECT ip FROM EXTERNAL_PIHOLE.network_addresses WHERE network_id = id ORDER BY lastseen DESC, ip LIMIT 1) as VARCHAR(100)) || ':' || cast( SUBSTR((SELECT name FROM EXTERNAL_PIHOLE.network_addresses WHERE network_id = id ORDER BY lastseen DESC, ip LIMIT 1), 0, INSTR((SELECT name FROM EXTERNAL_PIHOLE.network_addresses WHERE network_id = id ORDER BY lastseen DESC, ip LIMIT 1), '/')) as VARCHAR(100)) as Object_SecondaryID, datetime() as DateTime, macVendor as Watched_Value1, lastQuery as Watched_Value2, (SELECT name FROM EXTERNAL_PIHOLE.network_addresses WHERE network_id = id ORDER BY lastseen DESC, ip LIMIT 1) as Watched_Value3, 'null' as Watched_Value4, '' as Extra, hwaddr as ForeignKey FROM EXTERNAL_PIHOLE.network WHERE hwaddr NOT LIKE 'ip-%' AND hwaddr <> '00:00:00:00:00:00'; ", -> "options": [], -> "localized": ["name", "description"], -> "name" : [{ -> "language_code":"en_us", -> "string" : "SQL to run" -> }], -> "description": [{ -> "language_code":"en_us", -> "string" : "This SQL query is used to populate the coresponding UI tables under the Plugins section. This particular one selects data from a mapped PiHole SQLite database and maps it to the corresponding Plugin columns." -> }] -> } -> ``` - -## ๐Ÿ•ณ Filters - -Plugin entries can be filtered in the UI based on values entered into filter fields. The `txtMacFilter` textbox/field contains the Mac address of the currently viewed device, or simply a Mac address that's available in the `mac` query string (`?mac=aa:22:aa:22:aa:22:aa`). - - | Property | Required | Description | - |----------------------|----------------------|----------------------| - | `compare_column` | yes | Plugin column name that's value is used for comparison (**Left** side of the equation) | - | `compare_operator` | yes | JavaScript comparison operator | - | `compare_field_id` | yes | The `id` of a input text field containing a value is used for comparison (**Right** side of the equation)| - | `compare_js_template` | yes | JavaScript code used to convert left and right side of the equation. `{value}` is replaced with input values. | - | `compare_use_quotes` | yes | If `true` then the end result of the `compare_js_template` i swrapped in `"` quotes. Use to compare strings. | - - Filters are only applied if a filter is specified, and the `txtMacFilter` is not `undefined`, or empty (`--`). - -> ๐Ÿ”ŽExample: -> -> ```json -> "data_filters": [ -> { -> "compare_column" : "Object_PrimaryID", -> "compare_operator" : "==", -> "compare_field_id": "txtMacFilter", -> "compare_js_template": "'{value}'.toString()", -> "compare_use_quotes": true -> } -> ], -> ``` -> ->1. On the `pluginsCore.php` page is an input field with the id `txtMacFilter`: -> ->```html -> ->``` -> ->2. This input field is initialized via the `&mac=` query string. -> ->3. The app then proceeds to use this Mac value from this field and compares it to the value of the `Object_PrimaryID` database field. The `compare_operator` is `==`. -> ->4. Both values, from the database field `Object_PrimaryID` and from the `txtMacFilter` are wrapped and evaluated with the `compare_js_template`, that is `'{value}.toString()'`. -> ->5. `compare_use_quotes` is set to `true` so `'{value}'.toString()` is wrappe dinto `"` quotes. -> ->6. This results in for example this code: -> ->```javascript -> // left part of the expression coming from compare_column and right from the input field -> // notice the added quotes ()") around the left and right part of teh expression -> "eval('ac:82:ac:82:ac:82".toString()')" == "eval('ac:82:ac:82:ac:82".toString()')" ->``` -> - - -### ๐Ÿ—บ Mapping the plugin results into a database table - -Plugin results are always inserted into the standard `Plugin_Objects` database table. Optionally, NetAlertX can take the results of the plugin execution, and insert these results into an additional database table. This is enabled by with the property `"mapped_to_table"` in the `config.json` file. The mapping of the columns is defined in the `database_column_definitions` array. - -> [!NOTE] -> If results are mapped to the `CurrentScan` table, the data is then included into the regular scan loop, so for example notification for devices are sent out. - - ->๐Ÿ” Example: -> ->For example, this approach is used to implement the `DHCPLSS` plugin. The script parses all supplied "dhcp.leases" files, gets the results in the generic table format outlined in the "Column order and values" section above, takes individual values, and inserts them into the `CurrentScan` database table in the NetAlertX database. All this is achieved by: -> ->1. Specifying the database table into which the results are inserted by defining `"mapped_to_table": "CurrentScan"` in the root of the `config.json` file as shown below: -> ->```json ->{ -> "code_name": "dhcp_leases", -> "unique_prefix": "DHCPLSS", -> ... -> "data_source": "script", -> "localized": ["display_name", "description", "icon"], -> "mapped_to_table": "CurrentScan", -> ... ->} ->``` ->2. Defining the target column with the `mapped_to_column` property for individual columns in the `database_column_definitions` array of the `config.json` file. For example in the `DHCPLSS` plugin, I needed to map the value of the `Object_PrimaryID` column returned by the plugin, to the `cur_MAC` column in the NetAlertX database table `CurrentScan`. Notice the `"mapped_to_column": "cur_MAC"` key-value pair in the sample below. -> ->```json ->{ -> "column": "Object_PrimaryID", -> "mapped_to_column": "cur_MAC", -> "css_classes": "col-sm-2", -> "show": true, -> "type": "device_mac", -> "default_value":"", -> "options": [], -> "localized": ["name"], -> "name":[{ -> "language_code":"en_us", -> "string" : "MAC address" -> }] -> } ->``` -> ->3. That's it. The app takes care of the rest. It loops thru the objects discovered by the plugin, takes the results line-by-line, and inserts them into the database table specified in `"mapped_to_table"`. The columns are translated from the generic plugin columns to the target table columns via the `"mapped_to_column"` property in the column definitions. - -> [!NOTE] -> You can create a column mapping with a default value via the `mapped_to_column_data` property. This means that the value of the given column will always be this value. That also means that the `"column": "NameDoesntMatter"` is not important as there is no database source column. - - ->๐Ÿ” Example: -> ->```json ->{ -> "column": "NameDoesntMatter", -> "mapped_to_column": "cur_ScanMethod", -> "mapped_to_column_data": { -> "value": "DHCPLSS" -> }, -> "css_classes": "col-sm-2", -> "show": true, -> "type": "device_mac", -> "default_value":"", -> "options": [], -> "localized": ["name"], -> "name":[{ -> "language_code":"en_us", -> "string" : "MAC address" -> }] -> } ->``` - -#### params - -> [!IMPORTANT] -> An esier way to access settings in scripts is the `get_setting_value` method. -> ```python -> from helper import get_setting_value -> -> ... -> NTFY_TOPIC = get_setting_value('NTFY_TOPIC') -> ... -> -> ``` - -The `params` array in the `config.json` is used to enable the user to change the parameters of the executed script. For example, the user wants to monitor a specific URL. - -> ๐Ÿ”Ž Example: -> Passing user-defined settings to a command. Let's say, you want to have a script, that is called with a user-defined parameter called `urls`: -> -> ```bash -> root@server# python3 /app/front/plugins/website_monitor/script.py urls=https://google.com,https://duck.com -> ``` - -* You can allow the user to add URLs to a setting with the `function` property set to a custom name, such as `urls_to_check` (this is not a reserved name from the section "Supported settings `function` values" below). -* You specify the parameter `urls` in the `params` section of the `config.json` the following way (`WEBMON_` is the plugin prefix automatically added to all the settings): ```json { - "params" : [ - { - "name" : "urls", - "type" : "setting", - "value" : "WEBMON_urls_to_check" - }] + "data_filters": [ + { + "compare_column": "Object_PrimaryID", + "compare_operator": "==", + "compare_field_id": "txtMacFilter", + "compare_js_template": "'{value}'.toString()", + "compare_use_quotes": true + } + ] } ``` -* Then you use this setting as an input parameter for your command in the `CMD` setting. Notice `urls={urls}` in the below json: -```json - { - "function": "CMD", - "type": {"dataType":"string", "elements": [{"elementType" : "input", "elementOptions" : [] ,"transformers": []}]}, - "default_value":"python3 /app/front/plugins/website_monitor/script.py urls={urls}", - "options": [], - "localized": ["name", "description"], - "name" : [{ - "language_code":"en_us", - "string" : "Command" - }], - "description": [{ - "language_code":"en_us", - "string" : "Command to run" - }] - } -``` - -During script execution, the app will take the command `"python3 /app/front/plugins/website_monitor/script.py urls={urls}"`, take the `{urls}` wildcard and replace it with the value from the `WEBMON_urls_to_check` setting. This is because: - -1. The app checks the `params` entries -2. It finds `"name" : "urls"` -3. Checks the type of the `urls` params and finds `"type" : "setting"` -4. Gets the setting name from `"value" : "WEBMON_urls_to_check"` - - IMPORTANT: in the `config.json` this setting is identified by `"function":"urls_to_check"`, not `"function":"WEBMON_urls_to_check"` - - You can also use a global setting, or a setting from a different plugin -5. The app gets the user defined value from the setting with the code name `WEBMON_urls_to_check` - - let's say the setting with the code name `WEBMON_urls_to_check` contains 2 values entered by the user: - - `WEBMON_urls_to_check=['https://google.com','https://duck.com']` -6. The app takes the value from `WEBMON_urls_to_check` and replaces the `{urls}` wildcard in the setting where `"function":"CMD"`, so you go from: - - `python3 /app/front/plugins/website_monitor/script.py urls={urls}` - - to - - `python3 /app/front/plugins/website_monitor/script.py urls=https://google.com,https://duck.com` - -Below are some general additional notes, when defining `params`: - -- `"name":"name_value"` - is used as a wildcard replacement in the `CMD` setting value by using curly brackets `{name_value}`. The wildcard is replaced by the result of the `"value" : "param_value"` and `"type":"type_value"` combo configuration below. -- `"type":""` - is used to specify the type of the params, currently only 2 supported (`sql`,`setting`). - - `"type":"sql"` - will execute the SQL query specified in the `value` property. The sql query needs to return only one column. The column is flattened and separated by commas (`,`), e.g: `SELECT devMac from DEVICES` -> `Internet,74:ac:74:ac:74:ac,44:44:74:ac:74:ac`. This is then used to replace the wildcards in the `CMD` setting. - - `"type":"setting"` - The setting code name. A combination of the value from `unique_prefix` + `_` + `function` value, or otherwise the code name you can find in the Settings page under the Setting display name, e.g. `PIHOLE_RUN`. -- `"value": "param_value"` - Needs to contain a setting code name or SQL query without wildcards. -- `"timeoutMultiplier" : true` - used to indicate if the value should multiply the max timeout for the whole script run by the number of values in the given parameter. -- `"base64": true` - use base64 encoding to pass the value to the script (e.g. if there are spaces) +See [UI Components: Filters](PLUGINS_DEV_UI_COMPONENTS.md#filters) for full documentation. -> ๐Ÿ”ŽExample: -> -> ```json -> { -> "params" : [{ -> "name" : "ips", -> "type" : "sql", -> "value" : "SELECT devLastIP from DEVICES", -> "timeoutMultiplier" : true -> }, -> { -> "name" : "macs", -> "type" : "sql", -> "value" : "SELECT devMac from DEVICES" -> }, -> { -> "name" : "timeout", -> "type" : "setting", -> "value" : "NMAP_RUN_TIMEOUT" -> }, -> { -> "name" : "args", -> "type" : "setting", -> "value" : "NMAP_ARGS", -> "base64" : true -> }] -> } -> ``` +--- +## Database Mapping -#### โš™ Setting object structure - -> [!NOTE] -> The settings flow and when Plugin specific settings are applied is described under the [Settings system](./SETTINGS_SYSTEM.md). - -Required attributes are: - -| Property | Description | -| -------- | ----------- | -| `"function"` | Specifies the function the setting drives or a simple unique code name. See Supported settings function values for options. | -| `"type"` | Specifies the form control used for the setting displayed in the Settings page and what values are accepted. Supported options include: | -| | - `{"dataType":"string", "elements": [{"elementType" : "input", "elementOptions" : [{"type":"password"}] ,"transformers": ["sha256"]}]}` | -| `"localized"` | A list of properties on the current JSON level that need to be localized. | -| `"name"` | Displayed on the Settings page. An array of localized strings. See Localized strings below. | -| `"description"` | Displayed on the Settings page. An array of localized strings. See Localized strings below. | -| (optional) `"events"` | Specifies whether to generate an execution button next to the input field of the setting. Supported values: | -| | - `"test"` - For notification plugins testing | -| | - `"run"` - Regular plugins testing | -| (optional) `"override_value"` | Used to determine a user-defined override for the setting. Useful for template-based plugins, where you can choose to leave the current value or override it with the value defined in the setting. (Work in progress) | -| (optional) `"events"` | Used to trigger the plugin. Usually used on the `RUN` setting. Not fully tested in all scenarios. Will show a play button next to the setting. After clicking, an event is generated for the backend in the `Parameters` database table to process the front-end event on the next run. | - -### UI Component Types Documentation - -This section outlines the structure and types of UI components, primarily used to build HTML forms or interactive elements dynamically. Each UI component has a `"type"` which defines its structure, behavior, and rendering options. - -#### UI Component JSON Structure -The UI component is defined as a JSON object containing a list of `elements`. Each element specifies how it should behave, with properties like `elementType`, `elementOptions`, and any associated `transformers` to modify the data. The example below demonstrates how a component with two elements (`span` and `select`) is structured: +To import plugin data into NetAlertX tables for device discovery or notifications: ```json { - "function": "devIcon", - "type": { - "dataType": "string", - "elements": [ - { - "elementType": "span", - "elementOptions": [ - { "cssClasses": "input-group-addon iconPreview" }, - { "getStringKey": "Gen_SelectToPreview" }, - { "customId": "NEWDEV_devIcon_preview" } - ], - "transformers": [] - }, - { - "elementType": "select", - "elementHasInputValue": 1, - "elementOptions": [ - { "cssClasses": "col-xs-12" }, - { - "onChange": "updateIconPreview(this)" - }, - { "customParams": "NEWDEV_devIcon,NEWDEV_devIcon_preview" } - ], - "transformers": [] - } - ] - } + "mapped_to_table": "CurrentScan", + "database_column_definitions": [ + { + "column": "Object_PrimaryID", + "mapped_to_column": "cur_MAC", + "show": true, + "type": "device_mac", + "localized": ["name"], + "name": [{"language_code": "en_us", "string": "MAC Address"}] + } + ] } - ``` -### Rendering Logic +See [UI Components: Database Mapping](PLUGINS_DEV_UI_COMPONENTS.md#mapping-to-database-tables) for full documentation. -The code snippet provided demonstrates how the elements are iterated over to generate their corresponding HTML. Depending on the `elementType`, different HTML tags (like ``, `