Disabling cache to fix build issues 🩹

FIX unable to send mail with publisher mail plugin - thanks to @LouisOb 🙏

.github/workflows/docker_dev.yml

@@ -74,6 +74,10 @@ jobs:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
+      # # Disable this after use
+      # - name: Prune Docker Builder
+      #   run: docker builder prune --force
+
       - name: Build and push
         uses: docker/build-push-action@v3
         with:
@@ -82,5 +86,6 @@ jobs:
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-          cache-from: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache
-          cache-to: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache,mode=max
+          # # ⚠ disable cache if build is failing to download debian packages
+          # cache-from: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache
+          # cache-to: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache,mode=max

.github/workflows/docker_prod.yml

@@ -80,5 +80,6 @@ jobs:
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-          cache-from: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache
-          cache-to: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache,mode=max
+          # # ⚠ disable cache if build is failing to download debian packages
+          # cache-from: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache
+          # cache-to: type=registry,ref=ghcr.io/jokob-sk/pi.alert:buildcache,mode=max

README.md

@@ -1,6 +1,6 @@
-# 💻🔍 Network security scanner
+# 💻🔍 Network security scanner & notification framework
 
-Scans for devices, port changes on your WIFI/LAN and alerts you if unknown devices or changes are found.
+Get visibility of what's going on on your WIFI/LAN network. Scan for devices, port changes and get alerts if unknown devices or changes are found. Write your own [Plugins](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins#readme) with auto-generated UI and in-build notification system. 
 
 [![Docker](https://img.shields.io/github/actions/workflow/status/jokob-sk/Pi.Alert/docker_prod.yml?label=Build&logo=GitHub)](https://github.com/jokob-sk/Pi.Alert/actions/workflows/docker_prod.yml)
 [![GitHub Committed](https://img.shields.io/github/last-commit/jokob-sk/Pi.Alert?color=40ba12&label=Committed&logo=GitHub&logoColor=fff)](https://github.com/jokob-sk/Pi.Alert)
@@ -13,15 +13,15 @@ Scans for devices, port changes on your WIFI/LAN and alerts you if unknown devic
 
 ## Why PiAlert❓ 
 
-Most of us don't know what's going on on our home network, but we want our family and data to _be safe_.  _Command-line tools_ are great, but the output can be _hard to understand_ and action if you are not a network specialist 😖.
+Most of us don't know what's going on on our home network, but we want our family and data to be safe.  _Command-line tools_ are great, but the output can be _hard to understand_ and action if you are not a network specialist.
 
 PiAlert gives you peace of mind. _Visualize and immediately report 📬_ what is going on in your network - this is the first step to enhance your _network security 🔐_. 
 
-_PiAlert combines several network and other scanning tools 🔍 with notifications 📧 into one user-friendly package 📦_. You get an overview of network device Sessions, Connected devices, Favorites, Events, Presence, Down alerts, and IPs. You can schedule Nmap scans to detect changes in device ports and visualize your Network topology (even with undetectable, dummy devices). 
+_PiAlert combines several network and other scanning tools 🔍 with notifications 📧 into one user-friendly package 📦_. You get an overview of network device Sessions, Connected devices, Events, Presence, Down alerts, and IPs. You can schedule Nmap scans to detect changes in device ports and visualize your Network topology (even with undetectable, dummy devices).
 
 Setup a _kill switch ☠_ for your network via a smart plug with the available [Home Assistant](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/HOME_ASSISTANT.md) integration. Implement custom automations with the [CSV device Exports 📤](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins/csv_backup), [Webhooks](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/WEBHOOK_N8N.md), or [API endpoints](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/API.md) features. 
 
-Extend the app if you want to create your own scanner and handle the results and notifications in PiAlert. Check available [Plugins & Instructions](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins). 
+Extend the app if you want to create your own scanner [Plugin](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins#readme) and handle the results and notifications in PiAlert. 
 
 Looking forward to your contributions if you decide to share your work with the community ❤.
 
@@ -34,11 +34,10 @@ Looking forward to your contributions if you decide to share your work with the
 
 | Features    | Details    | 
 |-------------|-------------|
-|      🔍     |   The app scans your network for, **New devices**, **New connections** (re-connections), **Disconnections**, **"Always Connected" devices down**, Devices **IP changes** and **Internet IP address changes**. Discovery & scan methods include: **arp-scan**.  **Pi-hole - DB import**,  **Pi-hole - DHCP leases import**, **Generic DHCP leases import**. **UNIFI controller import**, **SNMP-enabled router import**. Check the [Plugins](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins) docs for more info on individual scans. |
+|      🔍     |   The app scans your network for, **New devices**, **New connections** (re-connections), **Disconnections**, **"Always Connected" devices down**, Devices **IP changes** and **Internet IP address changes**. Discovery & scan methods include: **arp-scan**.  **Pi-hole - DB import**,  **Pi-hole - DHCP leases import**, **Generic DHCP leases import**. **UNIFI controller import**, **SNMP-enabled router import**. Check the [Plugins](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins#readme) docs for more info on individual scans. |
 |📧           | Send notifications to more than 80+ services, including Telegram via [Apprise](https://hub.docker.com/r/caronc/apprise), or use [Pushsafer](https://www.pushsafer.com/), or [NTFY](https://ntfy.sh/). |
 |🧩           | Feed your data and device changes into [Home Assistant](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/HOME_ASSISTANT.md), read [API endpoints](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/API.md), or use [Webhooks](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/WEBHOOK_N8N.md) to setup custom automation flows.  |
-|➕           | Build your own scanners with the [Plugin system](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins) |
-
+|➕           | Build your own scanners with the [Plugin system](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins#readme) |
 
 
 ## Installation & Documentation

back/update_vendors.sh

@@ -1,4 +1,5 @@
-#!/bin/sh
+#!/usr/bin/env bash
+
 # ------------------------------------------------------------------------------
 #  Pi.Alert
 #  Open Source Network Guard / WIFI & LAN intrusion detector 
@@ -20,15 +21,15 @@ echo "---------------------------------------------------------"
 
 # ----------------------------------------------------------------------
 echo Updating... /usr/share/ieee-data/
-cd /usr/share/ieee-data/
+cd /usr/share/ieee-data/ || { echo "could not enter /usr/share/ieee-data directory"; exit 1; }
 
 sudo mkdir -p 2_backup
-sudo cp *.txt 2_backup
-sudo cp *.csv 2_backup
+sudo cp -- *.txt 2_backup
+sudo cp -- *.csv 2_backup
 echo ""
 echo Download Start
 echo ""
-sudo curl $1  -LO https://standards-oui.ieee.org/iab/iab.csv \
+sudo curl "$1"  -LO https://standards-oui.ieee.org/iab/iab.csv \
               -LO https://standards-oui.ieee.org/iab/iab.txt \
               -LO https://standards-oui.ieee.org/oui28/mam.csv \
               -LO https://standards-oui.ieee.org/iab/iab.txt \
@@ -44,10 +45,10 @@ echo Download Finished
 # ----------------------------------------------------------------------
 echo ""
 echo Updating... /usr/share/arp-scan/
-cd /usr/share/arp-scan
+cd /usr/share/arp-scan || { echo "could not enter /usr/share/arp-scan directory"; exit 1; }
 
 sudo mkdir -p 2_backup
-sudo cp *.txt 2_backup
+sudo cp -- *.txt 2_backup
 
 # Update from /usb/lib/ieee-data
 sudo get-iab -v

docker-compose.yml

@@ -11,10 +11,10 @@ services:
     network_mode: host
     # restart: unless-stopped
     volumes:
-      - ${APP_DATA_LOCATION}/pialert_dev/config:/home/pi/pialert/config
-      # - ${APP_DATA_LOCATION}/pialert/config:/home/pi/pialert/config
-      - ${APP_DATA_LOCATION}/pialert_dev/db:/home/pi/pialert/db      
-      # - ${APP_DATA_LOCATION}/pialert/db:/home/pi/pialert/db           
+      # - ${APP_DATA_LOCATION}/pialert_dev/config:/home/pi/pialert/config
+      - ${APP_DATA_LOCATION}/pialert/config:/home/pi/pialert/config
+      # - ${APP_DATA_LOCATION}/pialert_dev/db:/home/pi/pialert/db      
+      - ${APP_DATA_LOCATION}/pialert/db:/home/pi/pialert/db           
       # (optional) useful for debugging if you have issues setting up the container
       - ${LOGS_LOCATION}:/home/pi/pialert/front/log      
       # ---------------------------------------------------------------------------

dockerfiles/README.md

@@ -4,7 +4,7 @@
 [![Docker Pulls](https://img.shields.io/docker/pulls/jokobsk/pi.alert?label=Pulls&logo=docker&color=0aa8d2&logoColor=fff)](https://hub.docker.com/r/jokobsk/pi.alert)
 [![Docker Pushed](https://img.shields.io/badge/dynamic/json?color=0aa8d2&logoColor=fff&label=Pushed&query=last_updated&url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fjokobsk%2Fpi.alert%2F&logo=docker&link=http://left&link=https://hub.docker.com/repository/docker/jokobsk/pi.alert)](https://hub.docker.com/r/jokobsk/pi.alert)
 
-# PiAlert 💻🔍 Network security scanner
+# PiAlert 💻🔍 Network security scanner & notification framework
 
   | 🐳 [Docker hub](https://registry.hub.docker.com/r/jokobsk/pi.alert) |  📑 [Docker guide](https://github.com/jokob-sk/Pi.Alert/blob/main/dockerfiles/README.md) |🆕 [Release notes](https://github.com/jokob-sk/Pi.Alert/releases) | 📚 [All Docs](https://github.com/jokob-sk/Pi.Alert/tree/main/docs) |
   |----------------------|----------------------| ----------------------|  ----------------------| 
@@ -82,6 +82,19 @@ There are 2 approaches how to get PiHole devices imported. Via the PiHole import
 
 > [!NOTE]
 > It's recommended to use the same schedule interval for all plugins responsible for discovering new devices.
+
+
+#### 🧭 Community guides
+
+> Primarily use the official installation guides in this document and use community content as suplementary material. Open an issue if you'd like to add your link to the list 🙏 
+
+- 📄 [How to Install Pi.Alert on Your Synology NAS - Marius hosting (English)](https://mariushosting.com/how-to-install-pi-alert-on-your-synology-nas/) (Updated frequently)
+- 📄 [시놀/헤놀에서 네트워크 스캐너 Pi.Alert Docker로 설치 및 사용하기 (Korean)](https://blog.dalso.org/article/%EC%8B%9C%EB%86%80-%ED%97%A4%EB%86%80%EC%97%90%EC%84%9C-%EB%84%A4%ED%8A%B8%EC%9B%8C%ED%81%AC-%EC%8A%A4%EC%BA%90%EB%84%88-pi-alert-docker%EB%A1%9C-%EC%84%A4%EC%B9%98-%EB%B0%8F-%EC%82%AC%EC%9A%A9) (July 2023)
+- ▶ [Pi.Alert auf Synology & Docker by - Jürgen Barth (German)](https://www.youtube.com/watch?v=-ouvA2UNu-A) (March 2023)
+- ▶ [Top Docker Container for Home Server Security - VirtualizationHowto (English)](https://www.youtube.com/watch?v=tY-w-enLF6Q) (March 2023)
+- ▶ [Pi.Alert or WatchYourLAN can alert you to unknown devices appearing on your WiFi or LAN network - Danie van der Merwe (English)](https://www.youtube.com/watch?v=v6an9QG2xF0) (November 2022)
+
+> Ordered by last update time. 
  
 ### **Common issues** 
 
@@ -89,7 +102,10 @@ There are 2 approaches how to get PiHole devices imported. Via the PiHole import
 
 ⚠ Check also common issues and [debugging tips](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/DEBUG_TIPS.md). 
 
-## 📄 Examples
+> [!NOTE]
+> You can bulk-update devices via the [CSV import method](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/DEVICES_BULK_EDITING.md).
+
+## 📄 docker-compose.yml Examples
 
 ### Example 1
 

dockerfiles/start.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 echo "---------------------------------------------------------"
 echo "[INSTALL]                                    Run start.sh"
@@ -7,6 +7,12 @@ echo "---------------------------------------------------------"
 
 INSTALL_DIR=/home/pi  # Specify the installation directory here
 
+# DO NOT CHANGE ANYTHING BELOW THIS LINE!
+WEB_UI_DIR=/var/www/html/pialert
+NGINX_CONFIG_FILE=/etc/nginx/conf.d/pialert.conf
+OUI_FILE="/usr/share/arp-scan/ieee-oui.txt" # Define the path to ieee-oui.txt and ieee-iab.txt
+# DO NOT CHANGE ANYTHING ABOVE THIS LINE!
+
 # if custom variables not set we do not need to do anything
 if [ -n "${TZ}" ]; then    
   FILECONF=$INSTALL_DIR/pialert/config/pialert.conf 
@@ -29,38 +35,50 @@ echo "[INSTALL] Run setup scripts"
 "$INSTALL_DIR/pialert/dockerfiles/user-mapping.sh"
 "$INSTALL_DIR/pialert/install/install_dependencies.sh" # if modifying this file transfer the chanegs into the root Dockerfile as well!
 
-# Change port number if set
-if [ -n "${PORT}" ]; then  
-  sed -ie 's/listen 20211/listen '${PORT}'/g' /etc/nginx/sites-available/default
-fi 
-
 echo "[INSTALL] Setup NGINX"
 
-# Remove /html folder if exists
-sudo rm -R /var/www/html 
+# Remove default NGINX site if it is symlinked, or backup it otherwise
+if [ -L /etc/nginx/sites-enabled/default ] ; then
+  echo "Disabling default NGINX site, removing sym-link in /etc/nginx/sites-enabled"
+  sudo rm /etc/nginx/sites-enabled/default
+elif [ -f /etc/nginx/sites-enabled/default ]; then
+  echo "Disabling default NGINX site, moving config to /etc/nginx/sites-available"
+  sudo mv /etc/nginx/sites-enabled/default /etc/nginx/sites-available/default.bkp_pialert
+fi
+
+# Clear existing directories and files
+if [ -d $WEB_UI_DIR ]; then
+  echo "Removing existing PiAlert web-UI"
+  sudo rm -R $WEB_UI_DIR
+fi
+
+if [ -f $NGINX_CONFIG_FILE ]; then
+  echo "Removing existing PiAlert NGINX config"
+  sudo rm $NGINX_CONFIG_FILE
+fi
 
 # create symbolic link to the pialert install directory
-ln -s $INSTALL_DIR/pialert/front /var/www/html 
-# remove dfault NGINX site
-sudo rm /etc/nginx/sites-available/default
+ln -s $INSTALL_DIR/pialert/front $WEB_UI_DIR
 # create symbolic link to NGINX configuaration coming with PiAlert
-sudo ln -s "$INSTALL_DIR/pialert/install/default" /etc/nginx/sites-available/default
-# use user-supplied port
-sudo sed -i 's/listen 80/listen '"$PORT"'/g' /etc/nginx/sites-available/default
+sudo ln -s "$INSTALL_DIR/pialert/install/pialert.conf" /etc/nginx/conf.d/pialert.conf
+
+# Use user-supplied port if set
+if [ -n "${PORT}" ]; then
+  echo "Setting webserver to user-supplied port ($PORT)"
+  sudo sed -i 's/listen 20211/listen '"$PORT"'/g' /etc/nginx/conf.d/pialert.conf
+fi
 
 # Change web interface address if set
-if [ -n "${LISTEN_ADDR}" ]; then  
-  sed -ie 's/listen /listen '${LISTEN_ADDR}:'/g' /etc/nginx/sites-available/default
+if [ -n "${LISTEN_ADDR}" ]; then
+  echo "Setting webserver to user-supplied address ($LISTEN_ADDR)"
+  sed -ie 's/listen /listen '"${LISTEN_ADDR}":'/g' /etc/nginx/conf.d/pialert.conf
 fi
 
 # Run the hardware vendors update at least once
 echo "[INSTALL] Run the hardware vendors update"
 
-# Define the path to ieee-oui.txt and ieee-iab.txt
-oui_file="/usr/share/arp-scan/ieee-oui.txt"
-
 # Check if ieee-oui.txt or ieee-iab.txt exist
-if [ -f "$oui_file" ]; then
+if [ -f "$OUI_FILE" ]; then
   echo "The file ieee-oui.txt exists. Skipping update_vendors.sh..."
 else
   echo "The file ieee-oui.txt does not exist. Running update_vendors..."
@@ -77,7 +95,7 @@ fi
 echo "[INSTALL] Fixing file permissions"
 
 
-chmod -R a+rwx /var/www/html
+chmod -R a+rwx $WEB_UI_DIR
 chmod -R a+rw $INSTALL_DIR/pialert/front/log
 chmod -R a+rwx $INSTALL_DIR
 

dockerfiles/user-mapping.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 echo "---------------------------------------------------------"
 echo "[INSTALL]                             Run user-mapping.sh"
@@ -9,7 +9,7 @@ if [ -z "${USER}" ]; then
 fi
 
 # if both not set we do not need to do anything
-if [ -z "${HOST_USER_ID}" -a -z "${HOST_USER_GID}" ]; then
+if [ -z "${HOST_USER_ID}" ] && [ -z "${HOST_USER_GID}" ]; then
     echo "Nothing to do here." ; exit 0
 fi
 
@@ -20,20 +20,20 @@ USER_GID=${HOST_USER_GID:=$USER_GID}
 
 LINE=$(grep -F "${USER}" /etc/passwd)
 # replace all ':' with a space and create array
-array=( ${LINE//:/ } )
+array=( "${LINE//:/ }" )
 
 # home is 5th element
 USER_HOME=${array[4]}
 
 # print debug output
-echo  USER_ID  : ${USER_ID};
-echo  USER_GID : ${USER_GID};
-echo  USER_HOME: ${USER_HOME};
-echo  TZ       : ${TZ};
+echo  USER_ID"  ": "${USER_ID}";
+echo  USER_GID : "${USER_GID}";
+echo  USER_HOME: "${USER_HOME}";
+echo  TZ"       ": "${TZ}";
 
 sed -i -e "s/^${USER}:\([^:]*\):[0-9]*:[0-9]*/${USER}:\1:${USER_ID}:${USER_GID}/"  /etc/passwd
 sed -i -e "s/^${USER}:\([^:]*\):[0-9]*/${USER}:\1:${USER_GID}/"  /etc/group
 
-chown -R ${USER_ID}:${USER_GID} ${USER_HOME}
+chown -R "${USER_ID}:${USER_GID} ${USER_HOME}"
 
-exec su - "${USER}"
+exec su - "${USER}"

docs/HW_INSTALL.md

@@ -1,12 +1,37 @@
 # How to install PiAlert on the server hardware
 
-To download and install PiAlert on the hardware/server directly use `curl` or `wget` commands. 
+To download and install PiAlert on the hardware/server directly use `curl` or `wget` commands.
 
-> [!NOTE] 
-> This is an Experimental feature 🧪 and it relies on community support. 
+> [!NOTE]
+> This is an Experimental feature 🧪 and it relies on community support.
+>
+> There is no guarantee that the install script or any other script will gracefully handle other installed software.
+> Data loss is a possibility, **it is recommended to install PiAlert using the supplied Docker image**.
+
+A warning to the installation method below: Piping to bash is [controversial](https://pi-hole.net/2016/07/25/curling-and-piping-to-bash) and may
+be dangerous, as you cannot see the code that's about to be executed on your system.
+
+Alternatively you can download the installation script `install/install.sh` from the repository and check the code yourself (beware other scripts are
+downloaded too - only from this repo).
 
 PiAlert will be installed in `home/pi/pialert/` and run on port number `20211`.
 
+Some facts about what and where something will be changed/installed by the HW install setup (may not contain everything!):
+
+- `/home/pi/pialert` directory will be deleted and newly created
+- `/home/pi/pialert` will contain the whole repository (downloaded by `install/install.sh`)
+- The default NGINX site `/etc/nginx/sites-enabled/default` will be disabled (sym-link deleted or backed up to `sites-available`)
+- `/var/www/html/pialert` directory will be deleted and newly created
+- `/etc/nginx/conf.d/pialert.conf` will be sym-linked to `/home/pi/pialert/install/pialert.conf`
+- Some files (IEEE device vendors info, ...) will be created in the directory where the installation script is executed
+
+## Limitations
+
+- No system service is provided. PiAlert must be started using `/home/pi/pialert/dockerfiles/start.sh`.
+- No checks for other running software is done.
+- Only tested to work on Debian Bookworm (Debian 12).
+- **EXPERIMENTAL** and not recommended way to install PiAlert.
+
 ## CURL
 
 ```bash
@@ -15,11 +40,10 @@ curl -o install.sh https://raw.githubusercontent.com/jokob-sk/Pi.Alert/main/inst
 
 ## WGET
 
-
 ```bash
 wget https://raw.githubusercontent.com/jokob-sk/Pi.Alert/main/install/install.sh -O install.sh && sudo chmod +x install.sh && sudo ./install.sh
 ```
 
 These commands will download the `install.sh` script from the GitHub repository, make it executable with `chmod`, and then run it using `./install.sh`.
 
-Make sure you have the necessary permissions to execute the script.
+Make sure you have the necessary permissions to execute the script.

docs/SETTINGS_SYSTEM.md

@@ -10,7 +10,7 @@ The source of truth for user-defined values is the `pialert.conf` file. Editing
 
 #### Settings database table
 
-The `Settings` database table contains settings for App run purposes. The table is recreated every time the App restarts. The settings are loaded from the source-of-truth, that is the `pialert.conf` file. A high-level overview on the databse structure can be found in the [database documentation](/docs/DATABASE.md). 
+The `Settings` database table contains settings for App run purposes. The table is recreated every time the App restarts. The settings are loaded from the source-of-truth, that is the `pialert.conf` file. A high-level overview on the database structure can be found in the [database documentation](/docs/DATABASE.md). 
 
 #### table_settings.json
 

front/deviceDetails.php

@@ -644,14 +644,15 @@ if ($ENABLED_DARKMODE === True) {
   // ------------------------------------------------------------
   function getDevicesList()
   {
-    // Read cache
-    devicesList = getCache('devicesList');
+    // Read cache (skip cookie expiry check)
+    devicesList = getCache('devicesListAll_JSON', true);
     
     if (devicesList != '') {
         devicesList = JSON.parse (devicesList);
     } else {
         devicesList = [];
     }
+
     return devicesList;
   }
 
@@ -1283,7 +1284,7 @@ function getDeviceData (readAllData=false) {
         history.pushState(null, '', newRelativePathQuery);
         getSessionsPresenceEvents();
         
-        devicesList = getDevicesList();
+        devicesList = getDevicesList();        
 
         $('#txtMAC').val                             (deviceData['dev_MAC']);
         $('#txtName').val                            (deviceData['dev_Name']);
@@ -1324,7 +1325,8 @@ function getDeviceData (readAllData=false) {
       }
 
       // Check if device is part of the devicesList      
-      pos = devicesList.findIndex(item => item.rowid == deviceData['rowid']);      
+      pos = devicesList.findIndex(item => item.rowid == deviceData['rowid']);          
+      
       if (pos == -1) {
         devicesList.push({"rowid" : deviceData['rowid'], "mac" : deviceData['dev_MAC'], "name": deviceData['dev_Name'], "type": deviceData['dev_DeviceType']});
         pos=0;
@@ -1398,7 +1400,7 @@ function performSwitch(direction)
   // get new mac from the devicesList. Don't change to the commented out line below, the mac query string in the URL isn't updated yet!
   // mac = params.mac;
   
-  mac = devicesList[pos].mac.toString();
+  mac = devicesList[pos].dev_MAC.toString();
 
   setCache("piaDeviceDetailsMac", mac);
     
@@ -1457,6 +1459,9 @@ function setDeviceData (direction='', refreshCallback='') {
     window.onbeforeunload = null;
     somethingChanged      = false;
 
+    // refresh API
+    updateApi()
+
     // Callback fuction
     if (typeof refreshCallback == 'function') {
       refreshCallback(direction);
@@ -1464,6 +1469,25 @@ function setDeviceData (direction='', refreshCallback='') {
   });
 }
 
+// --------------------------------------------------------
+// Calls a backend function to add a front-end event to an execution queue
+function updateApi()
+{
+
+  // value has to be in format event|param. e.g. run|ARPSCAN
+  action = `update_api|devices`
+
+  $.ajax({
+    method: "POST",
+    url: "php/server/util.php",
+    data: { function: "addToExecutionQueue", action: action  },
+    success: function(data, textStatus) {
+        console.log(data)
+    }
+  })
+}
+
+
 // -----------------------------------------------------------------------------
 function askSkipNotifications () {
   // Check MAC
@@ -1630,36 +1654,10 @@ function deleteDevice () {
 
   // Deactivate controls
   $('#panDetails :input').attr('disabled', true);
+
+  // refresh API
+  updateApi()
 }
-// -----------------------------------------------------------------------------
-function askDeleteDevice () {
-  // Check MAC
-  if (mac == '') {
-    return;
-  }
-
-  // Ask delete device
-  showModalWarning ('Delete Device', 'Are you sure you want to delete this device?<br>(maybe you prefer to archive it)',
-    '<?= lang('Gen_Cancel');?>', '<?= lang('Gen_Delete');?>', 'deleteDevice');
-}
-
-
-// -----------------------------------------------------------------------------
-function deleteDevice () {
-  // Check MAC
-  if (mac == '') {
-    return;
-  }
-
-  // Delete device
-  $.get('php/server/devices.php?action=deleteDevice&mac='+ mac, function(msg) {
-    showMessage (msg);
-  });
-
-  // Deactivate controls
-  $('#panDetails :input').attr('disabled', true);
-}
-
 
 // -----------------------------------------------------------------------------
 function getSessionsPresenceEvents () {
@@ -1812,8 +1810,8 @@ function toggleNetworkConfiguration(disable)
 
   if(disable)
   {       
-    $('#txtNetworkNodeMac').val(getString('Network_Root_Unconfigurable'));   
-    $('#txtNetworkPort').val(getString('Network_Root_Unconfigurable'));     
+  //   $('#txtNetworkNodeMac').val(getString('Network_Root_Unconfigurable'));   
+  //   $('#txtNetworkPort').val(getString('Network_Root_Unconfigurable'));     
     $('#txtNetworkPort').prop('readonly', true );
     $('.parentNetworkNode .input-group-btn').hide();
   }

front/devices.php

@@ -288,7 +288,12 @@ function main () {
           initializeDatatable();
 
           // query data
-          getDevicesTotals();          
+          getDevicesTotals();      
+          
+          // check if dat outdated and show spinner if so
+          handleLoadingDialog()
+
+
         });
       });
     });
@@ -390,7 +395,7 @@ function initializeDatatable (status) {
     }    
   }
 
-  $.get('api/table_devices.json', function(result) {           
+  $.get('api/table_devices.json?nocache=' + Date.now(), function(result) {           
     
     // Filter the data based on deviceStatus
     var filteredData = filterDataByStatus(result.data, deviceStatus);
@@ -641,21 +646,22 @@ function getDevicesTotals () {
 // -----------------------------------------------------------------------------
 function handleLoadingDialog()
 {
-  $.get('api/app_state.json?nocache=' + Date.now(), function(appState) {   
+  $.get('log/execution_queue.log?nocache=' + Date.now(), function(data) {   
     
-    console.log(appState["showSpinner"])
-    if(appState["showSpinner"])
-    { 
-      showSpinner("settings_old")
+
+    if(data.includes("update_api|devices"))
+    {       
+      showSpinner("devices_old")
 
       setTimeout("handleLoadingDialog()", 1000);
 
-    } else
+    } else if ($("#loadingSpinner").is(":visible"))
     {
-      hideSpinner()        
+      hideSpinner();     
+      location.reload();    
     }      
 
-    })
+  })
 
 }
 

front/plugins/README.md

@@ -400,7 +400,7 @@ Plugin results are always inserted into the standard `Plugin_Objects` database t
 >3.  That's it. PiAlert takes care of the rest. It loops thru the objects discovered by the plugin, takes the results line, by line and inserts them into the database table specified in `"mapped_to_table"`. The columns are translated from the generic plugin columns to the target table via the `"mapped_to_column"` property in the column definitions.
 
 > [!NOTE] 
-> You can create a column mapping with a default value via the `mapped_to_column_data` property. This means that the value of the given column will always be this value. Taht also menas that the `"column": "NameDoesntMatter"` is not important as there is no databse source column. 
+> You can create a column mapping with a default value via the `mapped_to_column_data` property. This means that the value of the given column will always be this value. Taht also menas that the `"column": "NameDoesntMatter"` is not important as there is no database source column. 
 
 
 >🔍 Example:

front/plugins/README_DE.md

@@ -381,7 +381,7 @@ Plugin results are always inserted into the standard `Plugin_Objects` database t
 >3.  That's it. PiAlert takes care of the rest. It loops thru the objects discovered by the plugin, takes the results line, by line and inserts them into the database table specified in `"mapped_to_table"`. The columns are translated from the generic plugin columns to the target table via the `"mapped_to_column"` property in the column definitions.
 
 > [!NOTE] 
-> You can create a column mapping with a default value via the `mapped_to_column_data` property. This means that the value of the given column will always be this value. Taht also menas that the `"column": "NameDoesntMatter"` is not important as there is no databse source column. 
+> You can create a column mapping with a default value via the `mapped_to_column_data` property. This means that the value of the given column will always be this value. Taht also menas that the `"column": "NameDoesntMatter"` is not important as there is no database source column. 
 
 
 >🔍 Beispiel:

front/plugins/_publisher_email/email_smtp.py

@@ -109,12 +109,12 @@ def send(pHTML, pText):
 
     if get_setting_value("LOG_LEVEL") == 'debug':
 
-        send_email(msg)
+        send_email(msg,smtp_timeout)
 
     else:
 
         try:
-            send_email(msg)
+            send_email(msg,smtp_timeout)
             
         except smtplib.SMTPAuthenticationError as e:            
             mylog('none', ['      ERROR: Couldn\'t connect to the SMTP server (SMTPAuthenticationError)'])
@@ -132,7 +132,7 @@ def send(pHTML, pText):
             mylog('none', ['      ERROR: ', str(e)])      
 
 # ----------------------------------------------------------------------------------
-def send_email(msg):
+def send_email(msg,smtp_timeout):
     # Send mail
     if get_setting_value('SMTP_FORCE_SSL'):
         mylog('debug', ['SMTP_FORCE_SSL == True so using .SMTP_SSL()'])

front/plugins/_publisher_mqtt/mqtt.py

@@ -183,7 +183,7 @@ def publish_sensor(client, sensorConfig):
     global mqtt_sensors   
 
     message = '{ \
-                "name":"'+ sensorConfig.deviceName +' '+sensorConfig.sensorName+'", \
+                "name":"'+sensorConfig.sensorName+'", \
                 "state_topic":"system-sensors/'+sensorConfig.sensorType+'/'+sensorConfig.deviceId+'/state", \
                 "value_template":"{{value_json.'+sensorConfig.sensorName+'}}", \
                 "unique_id":"'+sensorConfig.deviceId+'_sensor_'+sensorConfig.sensorName+'", \

front/plugins/dhcp_leases/script.py

@@ -11,7 +11,7 @@ import chardet
 sys.path.append("/home/pi/pialert/front/plugins")
 sys.path.append('/home/pi/pialert/pialert') 
 
-from plugin_helper import Plugin_Object, Plugin_Objects, handleEmpty
+from plugin_helper import Plugin_Object, Plugin_Objects, handleEmpty, is_mac
 from logger import mylog
 from dhcp_leases import DhcpLeases
 
@@ -76,16 +76,20 @@ def get_entries(path, plugin_objects):
             leases = DhcpLeases(path)
             leasesList = leases.get()
             for lease in leasesList:
-                plugin_objects.add_object(
-                    primaryId   = handleEmpty(lease.ethernet),    
-                    secondaryId = handleEmpty(lease.ip),  
-                    watched1    = handleEmpty(lease.active),   
-                    watched2    = handleEmpty(lease.hostname),
-                    watched3    = handleEmpty(lease.hardware),
-                    watched4    = handleEmpty(lease.binding_state),
-                    extra       = handleEmpty(path),
-                    foreignKey  = handleEmpty(lease.ethernet)
-                )
+
+                # filter out irrelevant entries (e.g. from OPNsense dhcp.leases files)
+                if is_mac(lease.ethernet):
+
+                    plugin_objects.add_object(
+                        primaryId   = handleEmpty(lease.ethernet),    
+                        secondaryId = handleEmpty(lease.ip),  
+                        watched1    = handleEmpty(lease.active),   
+                        watched2    = handleEmpty(lease.hostname),
+                        watched3    = handleEmpty(lease.hardware),
+                        watched4    = handleEmpty(lease.binding_state),
+                        extra       = handleEmpty(path),
+                        foreignKey  = handleEmpty(lease.ethernet)
+                    )
     return plugin_objects
 
 if __name__ == '__main__':    

front/plugins/pihole_scan/config.json

@@ -73,7 +73,7 @@
         }],
         "description": [{
             "language_code":"en_us",
-            "string" : "Specify when your PiHole device import from the PiHole databse will run. The typical setting would be <code>schedule</code> and then you specify a cron-like schedule in the <a href=\"#PIHOLE_RUN_SCHD\"><code>PIHOLE_RUN_SCHD</code>setting</a>. If enabled, you must map the pihole db into your container to the <code>:/etc/pihole/pihole-FTL.db</code> mount path as specified in the <code>DB_PATH</code> setting."
+            "string" : "Specify when your PiHole device import from the PiHole database will run. The typical setting would be <code>schedule</code> and then you specify a cron-like schedule in the <a href=\"#PIHOLE_RUN_SCHD\"><code>PIHOLE_RUN_SCHD</code>setting</a>. If enabled, you must map the pihole db into your container to the <code>:/etc/pihole/pihole-FTL.db</code> mount path as specified in the <code>DB_PATH</code> setting."
         },
         {
             "language_code":"es_es",

front/plugins/plugin_helper.py

@@ -45,6 +45,11 @@ def handleEmpty(input):
             input = re.sub(r'[^\x00-\x7F]+', ' ', input)
         return  input
 
+# -------------------------------------------------------------------
+# Check if a valid MAC address
+def is_mac(input):
+    return re.match("[0-9a-f]{2}([-:]?)[0-9a-f]{2}(\\1[0-9a-f]{2}){4}$", input.lower())
+
 # -------------------------------------------------------------------
 def decodeBase64(inputParamBase64):
 

front/plugins/snmp_discovery/script.py

@@ -13,7 +13,7 @@ import sys
 sys.path.append("/home/pi/pialert/front/plugins")
 sys.path.append('/home/pi/pialert/pialert') 
 
-from plugin_helper import Plugin_Object, Plugin_Objects, decodeBase64
+from plugin_helper import Plugin_Object, Plugin_Objects, decodeBase64, handleEmpty
 from logger import mylog
 from helper import timeNowTZ
 from const import logPath, pialertPath

front/plugins/unifi_import/script.py

@@ -32,6 +32,9 @@ LOCK_FILE = os.path.join(CUR_PATH, 'full_run.lock')
 
 requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
 
+
+pluginName = 'UNFIMP'
+
 # Workflow
 
 def main():
@@ -131,9 +134,15 @@ def get_entries(plugin_objects: Plugin_Objects) -> Plugin_Objects:
 
             name = set_name(name, hostName)
 
+            ipTmp = get_unifi_val(ap, 'ip')
+            
+            #  if IP not found use a default value
+            if ipTmp == "null":
+                ipTmp = '0.0.0.0'
+
             plugin_objects.add_object(
                 primaryId=ap['mac'],
-                secondaryId=get_unifi_val(ap, 'ip'),
+                secondaryId=ipTmp,
                 watched1=name,
                 watched2='Ubiquiti Networks Inc.',
                 watched3=deviceType,
@@ -175,6 +184,10 @@ def get_entries(plugin_objects: Plugin_Objects) -> Plugin_Objects:
                 if ipTmp == 'null':
                     ipTmp = get_unifi_val(user, 'fixed_ip')
 
+                #  if IP not found use a default value
+                if ipTmp == "null":
+                    ipTmp = '0.0.0.0'
+
                 plugin_objects.add_object(
                     primaryId=user['mac'],
                     secondaryId=ipTmp,
@@ -206,6 +219,7 @@ def get_unifi_val(obj, key):
     if res not in ['','None', None]:
         return res
 
+    mylog('debug', [f'[{pluginName}] Value not found for key "{key}" in obj "{json.dumps(obj)}"'])       
 
     return 'null'
 

front/pluginsCore.php

@@ -173,7 +173,7 @@ function processColumnValue(dbColumnDef, value, index, type) {
             
             for (const option of dbColumnDef.options) {
                 if (option.type === type) {
-                    console.log(option.param)
+                    // console.log(option.param)
                     value =  eval(option.param);
                 }
             }

install/install.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 echo "---------------------------------------------------------"
 echo "[INSTALL]                                  Run install.sh"
@@ -26,6 +26,10 @@ rm -R $INSTALL_DIR/pialert
 # Clone the application repository
 git clone https://github.com/jokob-sk/Pi.Alert "$INSTALL_DIR/pialert"
 
+# Check for buildtimestamp.txt existence, otherwise create it
+if [ ! -f $INSTALL_DIR/pialert/front/buildtimestamp.txt ]; then
+  date +%s > $INSTALL_DIR/pialert/front/buildtimestamp.txt
+fi
 
 # Start PiAlert
 "$INSTALL_DIR/pialert/dockerfiles/start.sh"

install/install_dependencies.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 echo "---------------------------------------------------------"
 echo "[INSTALL]                     Run install_dependencies.sh"

install/pialert.conf

@@ -1,6 +1,6 @@
 server {
-    listen 80 default_server;
-    root /var/www/html;
+    listen 20211 default_server;
+    root /var/www/html/pialert;
     index index.php;
     #rewrite /pialert/(.*) / permanent;
     add_header X-Forwarded-Prefix "/pialert" always;
@@ -15,4 +15,4 @@ server {
           fastcgi_send_timeout 600;
           fastcgi_read_timeout 600;
   }
-}
+}

pialert/__main__.py

@@ -123,7 +123,7 @@ def main ():
             # determine run/scan type based on passed time
             # --------------------------------------------
            
-            # Run splugin scripts which are set to run every timne after a scans finished            
+            # Runs plugin scripts which are set to run every timne after a scans finished            
             pluginsState = run_plugin_scripts(db,'always_after_scan', pluginsState)
 
             

pialert/database.py

@@ -24,7 +24,7 @@ class DB():
     def open (self):
         # Check if DB is open
         if self.sql_connection != None :
-            mylog('debug','openDB: databse already open')
+            mylog('debug','openDB: database already open')
             return
 
         mylog('none', '[Database] Opening DB' )
@@ -42,7 +42,7 @@ class DB():
     #-------------------------------------------------------------------------------
     def commitDB (self):
         if self.sql_connection == None :
-            mylog('debug','commitDB: databse is not open')
+            mylog('debug','commitDB: database is not open')
             return False
 
         # Commit changes to DB
@@ -57,7 +57,7 @@ class DB():
     #-------------------------------------------------------------------------------    
     def get_sql_array(self, query):
         if self.sql_connection == None :
-            mylog('debug','getQueryArray: databse is not open')
+            mylog('debug','getQueryArray: database is not open')
             return
 
         self.sql.execute(query)

pialert/device.py

@@ -200,17 +200,30 @@ def update_devices_data_from_scan (db):
                     WHERE NOT EXISTS (SELECT 1 FROM CurrentScan 
                                       WHERE dev_MAC = cur_MAC) """)
 
-    # Update IP & Vendor
-    mylog('debug', '[Update Devices] - 3 LastIP & Vendor')
+    # Update IP 
+    mylog('debug', '[Update Devices] - 3 LastIP ')
     sql.execute("""UPDATE Devices
                     SET dev_LastIP = (SELECT cur_IP FROM CurrentScan
-                                      WHERE dev_MAC = cur_MAC),
-                        dev_Vendor = (SELECT cur_Vendor FROM CurrentScan
-                                      WHERE dev_MAC = cur_MAC
-                                        )
+                                      WHERE dev_MAC = cur_MAC)
                     WHERE EXISTS (SELECT 1 FROM CurrentScan
                                   WHERE dev_MAC = cur_MAC) """)
 
+    # Update only devices with empty or NULL vendors
+    mylog('debug', '[Update Devices] - 3 Vendor')
+    sql.execute("""UPDATE Devices
+                    SET dev_Vendor = (
+                        SELECT cur_Vendor
+                        FROM CurrentScan
+                        WHERE dev_MAC = cur_MAC
+                    )
+                    WHERE 
+                        (dev_Vendor = "" OR dev_Vendor IS NULL)
+                        AND EXISTS (
+                            SELECT 1
+                            FROM CurrentScan
+                            WHERE dev_MAC = cur_MAC
+                        )""")
+
     # Update (unknown) or (name not found) Names if available
     mylog('debug','[Update Devices] - 4 Unknown Name')
     sql.execute ("""UPDATE Devices

pialert/helper.py

@@ -709,9 +709,9 @@ def checkNewVersion():
 
         dateTimeStr = data[0]["published_at"]
 
-        realeaseTimestamp = int(datetime.datetime.strptime(dateTimeStr, '%Y-%m-%dT%H:%M:%SZ').strftime('%s'))
+        releaseTimestamp = int(datetime.datetime.strptime(dateTimeStr, '%Y-%m-%dT%H:%M:%S%z').timestamp())
 
-        if realeaseTimestamp > buildTimestamp + 600:
+        if releaseTimestamp > buildTimestamp + 600:
             mylog('none', ["[Version check] New version of the container available!"])
             newVersion = True       
         else:

pialert/plugin.py

@@ -241,7 +241,7 @@ def execute_plugin(db, plugin, pluginsState = plugins_state() ):
                 if len(columns) == 9:
                     # Create a tuple containing values to be inserted into the database.
                     # Each value corresponds to a column in the table in the order of the columns.
-                    # must match the Plugins_Objects and Plugins_Events databse tables and can be used as input for the plugin_object_class.
+                    # must match the Plugins_Objects and Plugins_Events database tables and can be used as input for the plugin_object_class.
                     sqlParams.append(
                         (
                             0,                          # "Index" placeholder
@@ -281,7 +281,7 @@ def execute_plugin(db, plugin, pluginsState = plugins_state() ):
             if len(row) == 9 and (row[0] in ['','null']) == False :
                 # Create a tuple containing values to be inserted into the database.
                 # Each value corresponds to a column in the table in the order of the columns.
-                # must match the Plugins_Objects and Plugins_Events databse tables and can be used as input for the plugin_object_class
+                # must match the Plugins_Objects and Plugins_Events database tables and can be used as input for the plugin_object_class
                 sqlParams.append(
                     (
                         0,                          # "Index" placeholder
@@ -337,7 +337,7 @@ def execute_plugin(db, plugin, pluginsState = plugins_state() ):
             if len(row) == 9 and (row[0] in ['','null']) == False :
                 # Create a tuple containing values to be inserted into the database.
                 # Each value corresponds to a column in the table in the order of the columns.
-                # must match the Plugins_Objects and Plugins_Events databse tables and can be used as input for the plugin_object_class
+                # must match the Plugins_Objects and Plugins_Events database tables and can be used as input for the plugin_object_class
                 sqlParams.append((
                     0,                            #  "Index" placeholder
                     plugin["unique_prefix"],      #  "Plugin" 
@@ -750,6 +750,9 @@ def check_and_run_user_event(db, pluginsState):
             pluginsState = handle_test(param, db, pluginsState)
         if event == 'run':
             pluginsState = handle_run(param, db, pluginsState)
+        if event == 'update_api':
+            # update API endpoints
+            update_api(db, False, param.split(','))  
 
     # Clear the log file
     open(logFile, "w").close()