MERGE: resolve conflicts

Signed-off-by: jokob-sk <jokob.sk@gmail.com>
This commit is contained in:
jokob-sk
2025-11-10 10:11:34 +11:00
77 changed files with 1670 additions and 811 deletions

View File

@@ -64,8 +64,9 @@ http://<server>:<GRAPHQL_PORT>/
* [Metrics](API_METRICS.md) Prometheus metrics and per-device status
* [Network Tools](API_NETTOOLS.md) Utilities like Wake-on-LAN, traceroute, nslookup, nmap, and internet info
* [Online History](API_ONLINEHISTORY.md) Online/offline device records
* [GraphQL](API_GRAPHQL.md) Advanced queries and filtering
* [GraphQL](API_GRAPHQL.md) Advanced queries and filtering for Devices, Settings and Language Strings
* [Sync](API_SYNC.md) Synchronization between multiple NetAlertX instances
* [Logs](API_LOGS.md) Purging of logs and adding to the event execution queue for user triggered events
* [DB query](API_DBQUERY.md) (⚠ Internal) - Low level database access - use other endpoints if possible
See [Testing](API_TESTS.md) for example requests and usage.

View File

@@ -1,9 +1,10 @@
# GraphQL API Endpoint
GraphQL queries are **read-optimized for speed**. Data may be slightly out of date until the file system cache refreshes. The GraphQL endpoints allows you to access the following objects:
GraphQL queries are **read-optimized for speed**. Data may be slightly out of date until the file system cache refreshes. The GraphQL endpoints allow you to access the following objects:
- Devices
- Settings
* Devices
* Settings
* Language Strings (LangStrings)
## Endpoints
@@ -190,11 +191,74 @@ curl 'http://host:GRAPHQL_PORT/graphql' \
}
```
---
## LangStrings Query
The **LangStrings query** provides access to localized strings. Supports filtering by `langCode` and `langStringKey`. If the requested string is missing or empty, you can optionally fallback to `en_us`.
### Sample Query
```graphql
query GetLangStrings {
langStrings(langCode: "de_de", langStringKey: "settings_other_scanners") {
langStrings {
langCode
langStringKey
langStringText
}
count
}
}
```
### Query Parameters
| Parameter | Type | Description |
| ---------------- | ------- | ---------------------------------------------------------------------------------------- |
| `langCode` | String | Optional language code (e.g., `en_us`, `de_de`). If omitted, all languages are returned. |
| `langStringKey` | String | Optional string key to retrieve a specific entry. |
| `fallback_to_en` | Boolean | Optional (default `true`). If `true`, empty or missing strings fallback to `en_us`. |
### `curl` Example
```sh
curl 'http://host:GRAPHQL_PORT/graphql' \
-X POST \
-H 'Authorization: Bearer API_TOKEN' \
-H 'Content-Type: application/json' \
--data '{
"query": "query GetLangStrings { langStrings(langCode: \"de_de\", langStringKey: \"settings_other_scanners\") { langStrings { langCode langStringKey langStringText } count } }"
}'
```
### Sample Response
```json
{
"data": {
"langStrings": {
"count": 1,
"langStrings": [
{
"langCode": "de_de",
"langStringKey": "settings_other_scanners",
"langStringText": "Other, non-device scanner plugins that are currently enabled." // falls back to en_us if empty
}
]
}
}
}
```
---
## Notes
* Device and settings queries can be combined in one request since GraphQL supports batching.
* Device, settings, and LangStrings queries can be combined in **one request** since GraphQL supports batching.
* The `fallback_to_en` feature ensures UI always has a value even if a translation is missing.
* Data is **cached in memory** per JSON file; changes to language or plugin files will only refresh after the cache detects a file modification.
* The `setOverriddenByEnv` flag helps identify setting values that are locked at container runtime.
* The schema is **read-only** — updates must be performed through other APIs or configuration management. See the other [API](API.md) endpoints for details.

179
docs/API_LOGS.md Normal file
View File

@@ -0,0 +1,179 @@
# Logs API Endpoints
Manage or purge application log files stored under `/app/log` and manage the execution queue. These endpoints are primarily used for maintenance tasks such as clearing accumulated logs or adding system actions without restarting the container.
Only specific, pre-approved log files can be purged for security and stability reasons.
---
## Delete (Purge) a Log File
* **DELETE** `/logs?file=<log_file>` → Purge the contents of an allowed log file.
**Query Parameter:**
* `file` → The name of the log file to purge (e.g., `app.log`, `stdout.log`)
**Allowed Files:**
```
app.log
app_front.log
IP_changes.log
stdout.log
stderr.log
app.php_errors.log
execution_queue.log
db_is_locked.log
```
**Authorization:**
Requires a valid API token in the `Authorization` header.
---
### `curl` Example (Success)
```sh
curl -X DELETE 'http://<server_ip>:<GRAPHQL_PORT>/logs?file=app.log' \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Accept: application/json'
```
**Response:**
```json
{
"success": true,
"message": "[clean_log] File app.log purged successfully"
}
```
---
### `curl` Example (Not Allowed)
```sh
curl -X DELETE 'http://<server_ip>:<GRAPHQL_PORT>/logs?file=not_allowed.log' \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Accept: application/json'
```
**Response:**
```json
{
"success": false,
"message": "[clean_log] File not_allowed.log is not allowed to be purged"
}
```
---
### `curl` Example (Unauthorized)
```sh
curl -X DELETE 'http://<server_ip>:<GRAPHQL_PORT>/logs?file=app.log' \
-H 'Accept: application/json'
```
**Response:**
```json
{
"error": "Forbidden"
}
```
---
## Add an Action to the Execution Queue
* **POST** `/logs/add-to-execution-queue` → Add a system action to the execution queue.
**Request Body (JSON):**
```json
{
"action": "update_api|devices"
}
```
**Authorization:**
Requires a valid API token in the `Authorization` header.
---
### `curl` Example (Success)
The below will update the API cache for Devices
```sh
curl -X POST 'http://<server_ip>:<GRAPHQL_PORT>/logs/add-to-execution-queue' \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' \
--data '{"action": "update_api|devices"}'
```
**Response:**
```json
{
"success": true,
"message": "[UserEventsQueueInstance] Action \"update_api|devices\" added to the execution queue."
}
```
---
### `curl` Example (Missing Parameter)
```sh
curl -X POST 'http://<server_ip>:<GRAPHQL_PORT>/logs/add-to-execution-queue' \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' \
--data '{}'
```
**Response:**
```json
{
"success": false,
"message": "Missing parameters",
"error": "Missing required 'action' field in JSON body"
}
```
---
### `curl` Example (Unauthorized)
```sh
curl -X POST 'http://<server_ip>:<GRAPHQL_PORT>/logs/add-to-execution-queue' \
-H 'Content-Type: application/json' \
--data '{"action": "update_api|devices"}'
```
**Response:**
```json
{
"error": "Forbidden"
}
```
---
## Notes
* Only predefined files in `/app/log` can be purged — arbitrary paths are **not permitted**.
* When a log file is purged:
* Its content is replaced with a short marker text: `"File manually purged"`.
* A backend log entry is created via `mylog()`.
* A frontend notification is generated via `write_notification()`.
* Execution queue actions are appended to `execution_queue.log` and can be processed asynchronously by background tasks or workflows.
* Unauthorized or invalid attempts are safely logged and rejected.
* For advanced log retrieval, analysis, or structured querying, use the frontend log viewer.
* Always ensure that sensitive or production logs are handled carefully — purging cannot be undone.

View File

@@ -83,7 +83,7 @@ You can also download the `app.conf` and `devices.csv` files from the **Maintena
### 📥 How to Restore
Map these files into your container as described in the [Setup documentation](https://github.com/jokob-sk/NetAlertX/blob/main/dockerfiles/README.md#docker-paths).
Map these files into your container as described in the [Setup documentation](./DOCKER_INSTALLATION.md).
---

View File

@@ -4,7 +4,7 @@
NetAlertX can be installed several ways. The best supported option is Docker, followed by a supervised Home Assistant instance, as an Unraid app, and lastly, on bare metal.
- [[Installation] Docker (recommended)](https://github.com/jokob-sk/NetAlertX/blob/main/dockerfiles/README.md)
- [[Installation] Docker (recommended)](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DOCKER_INSTALLATION.md)
- [[Installation] Home Assistant](https://github.com/alexbelgium/hassio-addons/tree/master/netalertx)
- [[Installation] Unraid App](https://unraid.net/community/apps)
- [[Installation] Bare metal (experimental - looking for maintainers)](https://github.com/jokob-sk/NetAlertX/blob/main/docs/HW_INSTALL.md)

View File

@@ -44,14 +44,19 @@ In Notification Processing settings, you can specify blanket rules. These allow
1. Notify on (`NTFPRCS_INCLUDED_SECTIONS`) allows you to specify which events trigger notifications. Usual setups will have `new_devices`, `down_devices`, and possibly `down_reconnected` set. Including `plugin` (dependenton the Plugin `<plugin>_WATCH` and `<plugin>_REPORT_ON` settings) and `events` (dependent on the on-device **Alert Events** setting) might be too noisy for most setups. More info in the [NTFPRCS plugin](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/notification_processing/README.md) on what events these selections include.
2. Alert down after (`NTFPRCS_alert_down_time`) is useful if you want to wait for some time before the system sends out a down notification for a device. This is related to the on-device **Alert down** setting and only devices with this checked will trigger a down notification.
3. A filter to allow you to set device-specific exceptions to New devices being added to the app.
4. A filter to allow you to set device-specific exceptions to generated Events.
## Ignoring devices 🔕
You can filter out unwanted notifications globally. This could be because of a misbehaving device (GoogleNest/GoogleHub (See also [ARPSAN docs and the `--exclude-broadcast` flag](https://github.com/jokob-sk/NetAlertX/tree/main/front/plugins/arp_scan#ip-flipping-on-google-nest-devices))) which flips between IP addresses, or because you want to ignore new device notifications of a certain pattern.
1. Events Filter (`NTFPRCS_event_condition`) - Filter out Events from notifications.
2. New Devices Filter (`NTFPRCS_new_dev_condition`) - Filter out New Devices from notifications, but log and keep a new device in the system.
## Ignoring devices 💻
![Ignoring new devices](./img/NOTIFICATIONS/NEWDEV_ignores.png)
You can completely ignore detected devices globally. This could be because your instance detects docker containers, you want to ignore devices from a specific manufacturer via MAC rules or you want to ignore devices on a specific IP range.
1. Ignored MACs (`NEWDEV_ignored_MACs`) - List of MACs to ignore.
2. Ignored IPs (`NEWDEV_ignored_IPs`) - List of IPs to ignore.
2. Ignored IPs (`NEWDEV_ignored_IPs`) - List of IPs to ignore.

View File

@@ -1,146 +1,192 @@
## config.json Lifecycle in NetAlertX
# Plugins Implementation Details
This document describes on a high level how `config.json` is read, processed, and used by the NetAlertX core and plugins. It also outlines the plugin output contract and the main plugin types.
Plugins provide data to the NetAlertX core, which processes it to detect changes, discover new devices, raise alerts, and apply heuristics.
> [!NOTE]
> For a deep-dive on the specific configuration options and sections of the `config.json` plugin manifest, consult the [Plugins Development Guide](PLUGINS_DEV.md).
---
## Overview: Plugin Data Flow
1. Each plugin runs on a defined schedule.
2. Aligning all plugin schedules is recommended so they execute in the same loop.
3. During execution, all plugins write their collected data into the **`CurrentScan`** table.
4. After all plugins complete, the `CurrentScan` table is evaluated to detect **new devices**, **changes**, and **triggers**.
Although plugins run independently, they contribute to the shared `CurrentScan` table.
To inspect its contents, set `LOG_LEVEL=trace` and check for the log section:
```
================ CurrentScan table content ================
```
---
## `config.json` Lifecycle
This section outlines how each plugins `config.json` manifest is read, validated, and used by the core and plugins.
It also describes plugin output expectations and the main plugin categories.
> [!TIP]
> For detailed schema and examples, see the [Plugin Development Guide](PLUGINS_DEV.md).
---
### 1. Loading
* On startup, the app core loads `config.json` for each plugin.
* The `config.json` represents a plugin manifest, that contains metadata and runtime settings.
* On startup, the core loads `config.json` for each plugin.
* The file acts as a **plugin manifest**, defining metadata, runtime configuration, and database mappings.
---
### 2. Validation
* The core checks that each required settings key (such as `RUN`) for a plugin exists.
* Invalid or missing values may be replaced with defaults, or the plugin may be disabled.
* The core validates required keys (for example, `RUN`).
* Missing or invalid entries may be replaced with defaults or cause the plugin to be disabled.
---
### 3. Preparation
* The plugins settings (paths, commands, parameters) are prepared.
* Database mappings (`mapped_to_table`, `database_column_definitions`) for data ingestion into the core app are parsed.
* Plugin parameters (paths, commands, and options) are prepared for execution.
* Database mappings (`mapped_to_table`, `database_column_definitions`) are parsed to define how data integrates with the main app.
---
### 4. Execution
* Plugins can be run at different core app execution points, such as on schedule, once on start, after a notification, etc.
* At runtime, the scheduler triggers plugins according to their `interval`.
* The plugin executes its command or script.
* Plugins may run:
* On a fixed schedule.
* Once at startup.
* After a notification or other trigger.
* The scheduler executes plugins according to their `interval`.
---
### 5. Parsing
* Plugin output is expected in **pipe (`|`)-delimited format**.
* The core parses lines into fields, matching the **plugin interface contract**.
* Plugin output must be **pipe-delimited (`|`)**.
* The core parses each output line following the **Plugin Interface Contract**, splitting and mapping fields accordingly.
---
### 6. Mapping
* Each parsed field is moved into the `Plugins_` database tables and can be mapped into a configured database table.
* Controlled by `database_column_definitions` and `mapped_to_table`.
* Example: `Object_PrimaryID → Devices.MAC`.
* Parsed fields are inserted into the plugins `Plugins_*` table.
* Data can be mapped into other tables (e.g., `Devices`, `CurrentScan`) as defined by:
* `database_column_definitions`
* `mapped_to_table`
**Example:** `Object_PrimaryID → devMAC`
---
### 6a. Plugin Output Contract
Each plugin must output results in the **plugin interface contract format**, pipe (`|`)-delimited values, in the column order described under [Plugin Interface Contract](PLUGINS_DEV.md)
All plugins must follow the **Plugin Interface Contract** defined in `PLUGINS_DEV.md`.
Output values are pipe-delimited in a fixed order.
#### IDs
#### Identifiers
* `Object_PrimaryID` and `Object_SecondaryID` identify the record (e.g. `MAC|IP`).
* `Object_PrimaryID` and `Object_SecondaryID` uniquely identify records (for example, `MAC|IP`).
#### **Watched values (`Watched_Value14`)**
#### Watched Values (`Watched_Value14`)
* Used by the core to detect changes between runs.
* Changes here can trigger **notifications**.
* Used by the core to detect changes between runs.
* Changes in these fields can trigger notifications.
#### **Extra value (`Extra`)**
#### Extra Field (`Extra`)
* Optional, extra field.
* Stored in the database but **not used for alerts**.
* Optional additional value.
* Stored in the database but not used for alerts.
#### **Helper values (`Helper_Value13`)**
#### Helper Values (`Helper_Value13`)
* Added for cases where more than IDs + watched + extra are needed.
* Can be made visible in the UI.
* Stored in the database but **not used for alerts**.
* Optional auxiliary data (for display or plugin logic).
* Stored but not alert-triggering.
#### **Mapping matters**
#### Mapping
* While the plugin output is free-form, the `database_column_definitions` and `mapped_to_table` settings in `config.json` determine the **target columns and data types** in NetAlertX.
* While the output format is flexible, the plugins manifest determines the destination and type of each field.
---
### 7. Persistence
* Data is upserted into the database.
* Conflicts are resolved using `Object_PrimaryID` + `Object_SecondaryID`.
* Parsed data is **upserted** into the database.
* Conflicts are resolved using the combined key: `Object_PrimaryID + Object_SecondaryID`.
---
### 8. Plugin Types and Expected Outputs
## Plugin Categories
Beyond the `data_source` setting, plugins fall into functional categories. Each has its own input requirements and output expectations:
Plugins fall into several functional categories depending on their purpose and expected outputs.
#### **Device discovery plugins**
### 1. Device Discovery Plugins
* **Inputs:** `N/A`, subnet, or API for discovery service, or similar.
* **Outputs:** At minimum `MAC` and `IP` that results in a new or updated device records in the `Devices` table.
* **Mapping:** Must be mapped to the `CurrentScan` table via `database_column_definitions` and `data_filters`.
* **Examples:** ARP-scan, NMAP device discovery (e.g., `ARPSCAN`, `NMAPDEV`).
#### **Device-data enrichment plugins**
* **Inputs:** Device identifier (usually `MAC`, `IP`).
* **Outputs:** Additional data for that device (e.g. open ports).
* **Mapping:** Controlled via `database_column_definitions` and `data_filters`.
* **Examples:** Ports, MQTT messages (e.g., `NMAP`, `MQTT`)
#### **Name resolver plugins**
* **Inputs:** Device identifiers (MAC, IP, or hostname).
* **Outputs:** Updated `devName` and `devFQDN` fields.
* **Mapping:** Not expected.
* **Note:** Currently requires **core app modification** to add new plugins, not fully driven by the plugins `config.json`.
* **Examples:** Avahiscan (e.g., `NBTSCAN`, `NSLOOKUP`).
#### **Generic plugins**
* **Inputs:** Whatever the script or query provides.
* **Outputs:** Data shown only in **Integrations → Plugins**, not tied to devices.
* **Mapping:** Not expected.
* **Examples:** External monitoring data (e.g., `INTRSPD`)
#### **Configuration-only plugins**
* **Inputs/Outputs:** None at runtime.
* **Mapping:** Not expected.
* **Examples:** Used to provide additional settings or execute scripts (e.g., `MAINT`, `CSVBCKP`).
* **Inputs:** None, subnet, or discovery API.
* **Outputs:** `MAC` and `IP` for new or updated device records in `Devices`.
* **Mapping:** Required usually into `CurrentScan`.
* **Examples:** `ARPSCAN`, `NMAPDEV`.
---
### 9. Post-Processing
### 2. Device Data Enrichment Plugins
* Notifications are generated if watched values change.
* UI is updated with new or updated records.
* All values that are configured to be shown in teh UI appear in the Plugins section.
* **Inputs:** Device identifiers (`MAC`, `IP`).
* **Outputs:** Additional metadata (for example, open ports or sensors).
* **Mapping:** Controlled via manifest definitions.
* **Examples:** `NMAP`, `MQTT`.
---
### 10. Summary
### 3. Name Resolver Plugins
The lifecycle of `config.json` entries is:
* **Inputs:** Device identifiers (`MAC`, `IP`, hostname`).
* **Outputs:** Updated `devName` and `devFQDN`.
* **Mapping:** Typically none.
* **Note:** Adding new resolvers currently requires a core change.
* **Examples:** `NBTSCAN`, `NSLOOKUP`.
---
### 4. Generic Plugins
* **Inputs:** Custom, based on the plugin logic or script.
* **Outputs:** Data displayed under **Integrations → Plugins** only.
* **Mapping:** Not required.
* **Examples:** `INTRSPD`, custom monitoring scripts.
---
### 5. Configuration-Only Plugins
* **Inputs/Outputs:** None at runtime.
* **Purpose:** Used for configuration or maintenance tasks.
* **Examples:** `MAINT`, `CSVBCKP`.
---
## Post-Processing
After persistence:
* The core generates notifications for any watched value changes.
* The UI updates with new or modified data.
* Plugins with UI-enabled data display under **Integrations → Plugins**.
---
## Summary
The lifecycle of a plugin configuration is:
**Load → Validate → Prepare → Execute → Parse → Map → Persist → Post-process**
Plugins must follow the **output contract**, and their category (discovery, specific, resolver, generic, config-only) defines what inputs they require and what outputs are expected.
Each plugin must:
* Follow the **output contract**.
* Declare its type and expected output structure.
* Define mappings and watched values clearly in `config.json`.

View File

@@ -13,7 +13,7 @@ There is also an in-app Help / FAQ section that should be answering frequently a
#### 🐳 Docker (Fully supported)
- The main installation method is as a [docker container - follow these instructions here](https://github.com/jokob-sk/NetAlertX/blob/main/dockerfiles/README.md).
- The main installation method is as a [docker container - follow these instructions here](./DOCKER_INSTALLATION.md).
#### 💻 Bare-metal / On-server (Experimental/community supported 🧪)

View File

@@ -15,7 +15,7 @@ The **Web UI** is served by an **nginx** server, while the **API backend** runs
APP_CONF_OVERRIDE={"GRAPHQL_PORT":"20212"}
```
For more information, check the [Docker installation guide](https://github.com/jokob-sk/NetAlertX/blob/main/dockerfiles/README.md).
For more information, check the [Docker installation guide](./DOCKER_INSTALLATION.md).
## Possible issues and troubleshooting

View File

@@ -21,7 +21,7 @@ The app can be installed different ways, with the best support of the docker-bas
NetAlertX is fully supported in Docker environments, allowing for easy setup and configuration. Follow the official guide to get started:
- [Docker Installation Guide](https://github.com/jokob-sk/NetAlertX/blob/main/dockerfiles/README.md)
- [Docker Installation Guide](./DOCKER_INSTALLATION.md)
This guide will take you through the process of setting up NetAlertX using Docker Compose or standalone Docker commands.