Update docker_prod.yml

Split up work 2023 05 30

.github/ISSUE_TEMPLATE/i-have-an-issue.md

@@ -12,35 +12,31 @@ assignees: ''
 
 **Paste last few lines from `pialert.log`**
 
+> You can use `tail -20 /home/pi/pialert/front/log/pialert.log`
+
 ```
-
-paste here
-
+paste_here
 ```
 
 **Paste your `pialert.conf` (remove personal info)**
 
 ```
-
-paste here
-
+paste_here
 ```
 
 **Paste your `docker-compose.yml` and `.env` (remove personal info)**
 
 `docker-compose.yml` 
-```
-
-paste here
 
 ```
-
+paste_here
+```
 
 `.env` 
-```
-
-paste here
 
 ```
+paste_here
+```
+
 **Screenshots**
 If applicable, add screenshots to help explain your problem.

.github/workflows/docker_prod.yml

@@ -46,6 +46,13 @@ jobs:
             type=ref,event=branch,suffix=-{{ sha }}
             type=ref,event=pr
             type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
+
+
+
+
+
+
+          
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
         uses: docker/login-action@v2

.gitignore

@@ -2,7 +2,12 @@
 .DS_Store
 config/pialert.conf
 db/*
+db/pialert.db
 front/log/*
 front/plugins/**/*.log
 **/%40eaDir/
 **/@eaDir/
+
+__pycache__/
+*.py[cod]
+*$py.class

Dockerfile

@@ -1,13 +1,14 @@
 FROM debian:bullseye-slim
 
 # default UID and GID
-ENV USER=pi USER_ID=1000 USER_GID=1000 TZ=Europe/London PORT=20211
+ENV USER=pi USER_ID=1000 USER_GID=1000  PORT=20211 
+#TZ=Europe/London
 
 # Todo, figure out why using a workdir instead of full paths don't work
 # Todo, do we still need all these packages? I can already see sudo which isn't needed
 
 RUN apt-get update \
-    && apt-get install --no-install-recommends tini ca-certificates curl libwww-perl arp-scan perl apt-utils cron sudo nginx-light php php-cgi php-fpm php-sqlite3 php-curl sqlite3 dnsutils net-tools python3 iproute2 nmap python3-pip zip -y \
+    && apt-get install --no-install-recommends tini snmp ca-certificates curl libwww-perl arp-scan perl apt-utils cron sudo nginx-light php php-cgi php-fpm php-sqlite3 php-curl sqlite3 dnsutils net-tools python3 iproute2 nmap python3-pip zip -y \
     && pip3 install requests paho-mqtt scapy cron-converter pytz json2table dhcp-leases pyunifi \
     && update-alternatives --install /usr/bin/python python /usr/bin/python3 10 \
     && apt-get clean autoclean \
@@ -46,3 +47,8 @@ RUN rm /etc/nginx/sites-available/default \
 ENTRYPOINT ["tini", "--"]
 
 CMD ["/home/pi/pialert/dockerfiles/start.sh"]
+
+
+
+
+## command to build docker:  DOCKER_BUILDKIT=1  docker build . --iidfile dockerID

README.md

@@ -51,13 +51,14 @@ The system continuously scans the network for, **New devices**, **New connection
     - Theme Selection (blue, red, green, yellow, black, purple) and Light/Dark-Mode Switch        
     - DB maintenance, Backup, Restore tools and CSV Export / Import
     - Simple login Support
-  - 🌟(Experimental) [Plugin system](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins)
+  - 🌟[Plugin system](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins)
     - Create custom plugins with automatically generated settings and UI. 
     - Monitor anything for changes
-    - Check the instructions carefully if you are up for a challenge! Current plugins include:
-      - Detecting Rogue DHCP servers
+    - Check the [instructions](https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins) carefully if you are up for a challenge! Current plugins include:
+      - Detecting Rogue DHCP servers via NMAP
       - Monitoring HTTP status changes of domains/URLs 
-      - Import devices from DHCP.leases files or a UniFi controller
+      - Import devices from DHCP.leases files, a UniFi controller, or an SNMP enabled router
+      - Creation of dummy devices to visualize your [network map](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/NETWORK_TREE.md)
 
   | ![Screen 1][screen1] | ![Screen 2][screen2] | ![Screen 5][screen5]  |
   |----------------------|----------------------| ----------------------| 
@@ -70,9 +71,10 @@ The system continuously scans the network for, **New devices**, **New connection
   - [WatchYourLAN](https://github.com/aceberg/WatchYourLAN) - Lightweight network IP scanner with web GUI (Open source)
   - [Fing](https://www.fing.com/) - Network scanner app for your Internet security (Commercial, Phone App, Proprietary hardware)
 
-### Old docs
+### 📚 Documentation
 
-Device Management [instructions](docs/DEVICE_MANAGEMENT.md) | Old Versions [History](docs/VERSIONS_HISTORY.md)
+- Initial Docker Setup: [Docker instructions](https://github.com/jokob-sk/Pi.Alert/blob/main/dockerfiles/README.md)
+- App Usage and Configuration: [All Documentation](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/README.md)
 
 ### License
   GPL 3.0 | [Read more here](LICENSE.txt) | Source of the [animated GIF (Loading Animation)](https://commons.wikimedia.org/wiki/File:Loading_Animation.gif) | Source of the [selfhosted Fonts](https://github.com/adobe-fonts/source-sans)
@@ -85,7 +87,8 @@ Device Management [instructions](docs/DEVICE_MANAGEMENT.md) | Old Versions [Hist
       - [leiweibau](https://github.com/leiweibau/Pi.Alert): Dark mode (and much more)
       - [Macleykun](https://github.com/Macleykun): Help with Dockerfile clean-up
       - [Final-Hawk](https://github.com/Final-Hawk): Help with NTFY, styling and other fixes
-      - [TeroRERO](https://github.com/terorero): Spanish translation      
+      - [TeroRERO](https://github.com/terorero): Spanish translation
+      - [Data-Monkey](https://github.com/Data-Monkey): Split-up of the python.py file and more     
       - Please see the [Git contributors](https://github.com/jokob-sk/Pi.Alert/graphs/contributors) for a full list of people and their contributions to the project
 
 ## ☕ Support me

docker-compose.yml

@@ -7,18 +7,21 @@ services:
     network_mode: "host"
     restart: unless-stopped
     volumes:
-      - ${APP_DATA_LOCATION}/pialert2/config:/home/pi/pialert/config
-      # - ${APP_DATA_LOCATION}/pialert/db/pialert.db:/home/pi/pialert/db/pialert.db
-      - ${APP_DATA_LOCATION}/pialert2/db:/home/pi/pialert/db      
+      - ${APP_DATA_LOCATION}/pialert_dev/config:/home/pi/pialert/config
+      # - ${APP_DATA_LOCATION}/pialert/config:/home/pi/pialert/config
+      - ${APP_DATA_LOCATION}/pialert_dev/db:/home/pi/pialert/db      
+      # - ${APP_DATA_LOCATION}/pialert/db:/home/pi/pialert/db           
       # (optional) useful for debugging if you have issues setting up the container
       - ${LOGS_LOCATION}:/home/pi/pialert/front/log      
+      # ---------------------------------------------------------------------------
       # DELETE START anyone trying to use this file: comment out / delete BELOW lines, they are only for development purposes
       - ${APP_DATA_LOCATION}/pialert/dhcp_samples/dhcp1.leases:/mnt/dhcp1.leases
       - ${APP_DATA_LOCATION}/pialert/dhcp_samples/dhcp2.leases:/mnt/dhcp2.leases
-      - ${DEV_LOCATION}/back/pialert.py:/home/pi/pialert/back/pialert.py
-      - ${DEV_LOCATION}/back/report_template.html:/home/pi/pialert/back/report_template.html
-      - ${DEV_LOCATION}/back/report_template_new_version.html:/home/pi/pialert/back/report_template_new_version.html
-      - ${DEV_LOCATION}/back/report_template.txt:/home/pi/pialert/back/report_template.txt
+      # - ${DEV_LOCATION}/back/pialert.py:/home/pi/pialert/back/pialert.py
+      - ${DEV_LOCATION}/pialert:/home/pi/pialert/pialert      
+      # - ${DEV_LOCATION}/back/report_template.html:/home/pi/pialert/back/report_template.html
+      # - ${DEV_LOCATION}/back/report_template_new_version.html:/home/pi/pialert/back/report_template_new_version.html
+      # - ${DEV_LOCATION}/back/report_template.txt:/home/pi/pialert/back/report_template.txt
       - ${DEV_LOCATION}/pholus:/home/pi/pialert/pholus
       - ${DEV_LOCATION}/dockerfiles:/home/pi/pialert/dockerfiles
       - ${APP_DATA_LOCATION}/pialert/php.ini:/etc/php/7.4/fpm/php.ini      
@@ -39,6 +42,7 @@ services:
       - ${DEV_LOCATION}/front/settings.php:/home/pi/pialert/front/settings.php
       - ${DEV_LOCATION}/front/plugins:/home/pi/pialert/front/plugins
       # DELETE END anyone trying to use this file: comment out / delete ABOVE lines, they are only for development purposes
+      # ---------------------------------------------------------------------------
     environment:
       - TZ=${TZ}
       - PORT=${PORT}

dockerfiles/README.md

@@ -53,28 +53,39 @@ docker run -d --rm --network=host \
 
 ### Config (`pialert.conf`)
 
-- The preferred wy is to manage the configuration via Settings 
-- YOu can modify [pialert.conf](https://github.com/jokob-sk/Pi.Alert/tree/main/config) directly if needed
-- ❗ To use the arp-scan method, you need to set the `SCAN_SUBNETS` variable. ()
+- If unavailable, the app generates a default `pialert.conf` and `pialert.db` file on the first run.
+- The preferred way is to manage the configuration via the Settings section in the UI.
+- You can modify [pialert.conf](https://github.com/jokob-sk/Pi.Alert/tree/main/config) directly, if needed.
+
+#### Important settings
+
+These are the most important settings to get at least some output in your Devices screen. Usually, only one approach is used, but you should be able to combine these approaches.
+
+##### For arp-scan: ENABLE_ARPSCAN, SCAN_SUBNETS
+
+- ❗ To use the arp-scan method, you need to set the `SCAN_SUBNETS` variable.
    * The adapter will probably be `eth0` or `eth1`. (Run `iwconfig` to find your interface name(s)) 
    * Specify the network filter (which **significantly** speeds up the scan process). For example, the filter `192.168.1.0/24` covers IP ranges 192.168.1.0 to 192.168.1.255.
    * Examples for one and two subnets  (❗ Note the `['...', '...']` format):
      * One subnet: `SCAN_SUBNETS    = ['192.168.1.0/24 --interface=eth0']`
-     * Two subnets:  `SCAN_SUBNETS    = ['192.168.1.0/24 --interface=eth0', '192.168.1.0/24 --interface=eth1']` 
-   * More documentation on how to [setup vlans](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SUBNETS.md) 
+     * Two subnets:  `SCAN_SUBNETS    = ['192.168.1.0/24 --interface=eth0', '192.168.1.0/24 --interface=eth1 -vlan=107']` 
+   * More documentation on how to e.g. [setup vlans & limitations](https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SUBNETS.md) 
 
+##### For pihole: PIHOLE_ACTIVE, DHCP_ACTIVE
 
-### 🛑 **Common issues** 
+* `PIHOLE_ACTIVE`: You need to map `:/etc/pihole/pihole-FTL.db in the docker-compose.yml` file if you enable this setting.
+* `DHCP_ACTIVE` : You need to map `:/etc/pihole/dhcp.leases in the docker-compose.yml` file if you enable this setting.
+
+### **Common issues** 
 
 💡 Before creating a new issue, please check if a similar issue was [already resolved](https://github.com/jokob-sk/Pi.Alert/issues?q=is%3Aissue+is%3Aclosed). 
 
 **Permissions**
 
 * If facing issues (AJAX errors, can't write to DB, empty screen, etc,) make sure permissions are set correctly, and check the logs under `/home/pi/pialert/front/log`. 
-* To solve permission issues you can also try to create a DB backup and then run a DB Restore via the **Maintenance > Backup/Restore** section.
-* You can try also setting the owner and group of the `pialert.db` by executing the following on the host system: `docker exec pialert chown -R www-data:www-data /home/pi/pialert/db/pialert.db`. 
+* To solve permission issues you can try setting the owner and group of the `pialert.db` by executing the following on the host system: `docker exec pialert chown -R www-data:www-data /home/pi/pialert/db/pialert.db`. 
 * Map to local User and Group IDs. Specify the enviroment variables `HOST_USER_ID` and `HOST_USER_GID` if needed.
-* Map the pialert.db file (⚠ not folder) to `:/home/pi/pialert/db/pialert.db` (see Examples below for details)
+* If still facing issues, try to map the pialert.db file (⚠ not folder) to `:/home/pi/pialert/db/pialert.db` (see Examples below for details)
 
 **Container restarts / crashes**
 

dockerfiles/start.sh

@@ -1,15 +1,15 @@
 #!/bin/sh
 /home/pi/pialert/dockerfiles/user-mapping.sh
 
-# if custom variables not set we do not need to do anything
-if [ -n "${TZ}" ]; then    
-  FILECONF=/home/pi/pialert/config/pialert.conf 
-  if [ -f "$FILECONF" ]; then
-    sed -ie "s|Europe/Berlin|${TZ}|g" /home/pi/pialert/config/pialert.conf 
-  else 
-    sed -ie "s|Europe/Berlin|${TZ}|g" /home/pi/pialert/back/pialert.conf_bak 
-  fi
-fi
+# # if custom variables not set we do not need to do anything
+# if [ -n "${TZ}" ]; then    
+#   FILECONF=/home/pi/pialert/config/pialert.conf 
+#   if [ -f "$FILECONF" ]; then
+#     sed -ie "s|Europe/Berlin|${TZ}|g" /home/pi/pialert/config/pialert.conf 
+#   else 
+#     sed -ie "s|Europe/Berlin|${TZ}|g" /home/pi/pialert/back/pialert.conf_bak 
+#   fi
+# fi
 
 if [ -n "${PORT}" ]; then  
   sed -ie 's/listen 20211/listen '${PORT}'/g' /etc/nginx/sites-available/default
@@ -28,4 +28,6 @@ chmod -R a+rw /home/pi/pialert/config
 /etc/init.d/nginx start
 
 # cron -f
-python /home/pi/pialert/back/pialert.py
+#python /home/pi/pialert/back/pialert.py
+# echo "[DEBUG] DATA MONKEY VERSION ..."
+python /home/pi/pialert/pialert/

docs/API.md

@@ -5,12 +5,7 @@ PiAlert comes with a simple API. These API endpoints are static files, that are
 
 ### When are the endpoints updated
 
-Once you enable the API (`ENABLE_API` setting), the endpoints are updated during these events:
-
-1) Always during a notification event.
-2) (optional) If `API_RUN` is set to `schedule` on a specified cron-like schedule specified by the `API_RUN_SCHD` setting.
-3) (optional) If `API_RUN` is set to `interval` every N seconds specified by the `API_RUN_INTERVAL` setting (minimum 5).
-
+Once you enable the API (`ENABLE_API` setting), the endpoints are updated when objects in the API endpoints are changed:
 
 ### Location of the endpoints
 

docs/DATABASE.md

@@ -0,0 +1,47 @@
+  
+  # A high-level description of the datbase structure
+
+  ⚠ Disclaimer: As I'm not the original author, some of the information might be inaccurate. Feel free to submit a PR to correct anything within this page or documentation in general. 
+
+  The MAC address is used as a foreign key in most cases. 
+
+  ## 🔍Tables overview
+  
+  | Table name | Description  | Sample data |
+  |----------------------|----------------------| ----------------------| 
+  | CurrentScan | Result of the current scan | ![Screen1][screen1]  |  
+  | Devices     | The main devices database that also contains the Network tree mappings. | ![Screen2][screen2]  | 
+  | DHCP_Leases | Used for importing devices from DHCP_Leases files. Also leveraged by some plugins. | ![Screen3][screen3]  | 
+  | Events | Used to collect connection/disconnection events. | ![Screen4][screen4]  | 
+  | Nmap_Scan | Contains results of the scheduled Nmap scan, taht is also displayed in the Nmap tab on each device. | ![Screen5][screen5]  | 
+  | Online_History   | Used to display the `Device presence over time` chart  | ![Screen6][screen6]  | 
+  | Parameters       | Used to pass values between the frontend and backend. | ![Screen7][screen7]  | 
+  | Pholus_Scan      | Scan results of the Pholus python network penetration script. | ![Screen8][screen8]  | 
+  | PiHole_Network   | Table to copy the devices from the PiHole database | ![Screen9][screen9]  | 
+  | Plugins_Events   | For capturing events exposed by a plugin via the `last_result.log` file. If unique then saved into the `Plugins_Objects` table. Entries are deleted once processed and stored in the `Plugins_History` and/or `Plugins_Objects` tables.  | ![Screen10][screen10]  | 
+  | Plugins_History  | History of all entries from the `Plugins_Events` table | ![Screen11][screen11]  | 
+  | Plugins_Language_Strings  | Language strings colelcted from the plugin `config.json` files used for string resolution in the frontend. | ![Screen12][screen12]  | 
+  | Plugins_Objects  | Unique objects detected by individual plugins. | ![Screen13][screen13]  | 
+  | ScanCycles  | (obsolete) Used to determine and identify different scan cycles. | ![Screen14][screen14]  | 
+  | Sessions  | Used to display sessions in the charts | ![Screen15][screen15]  | 
+  | Settings  | Database representation of the sum of all settings from `pialert.conf` and plugins coming from `config.json` files. | ![Screen16][screen16]  | 
+
+
+
+  [screen1]: /docs/img/DATABASE/CurrentScan.png
+  [screen2]: /docs/img/DATABASE/Devices.png
+  [screen3]: /docs/img/DATABASE/DHCP_Leases.png
+  [screen4]: /docs/img/DATABASE/Events.png
+  [screen5]: /docs/img/DATABASE/Nmap_Scan.png
+  [screen6]: /docs/img/DATABASE/Online_History.png
+  [screen7]: /docs/img/DATABASE/Parameters.png
+  [screen8]: /docs/img/DATABASE/Pholus_Scan.png
+  [screen9]: /docs/img/DATABASE/PiHole_Network.png
+  [screen10]: /docs/img/DATABASE/Plugins_Events.png
+  [screen11]: /docs/img/DATABASE/Plugins_History.png
+  [screen12]: /docs/img/DATABASE/Plugins_Language_Strings.png
+  [screen13]: /docs/img/DATABASE/Plugins_Objects.png
+  [screen14]: /docs/img/DATABASE/ScanCycles.png
+  [screen15]: /docs/img/DATABASE/Sessions.png
+  [screen16]: /docs/img/DATABASE/Settings.png
+

docs/DEVICE_MANAGEMENT.md

@@ -18,7 +18,7 @@ To edit device information:
   - **Owner**: Device owner (The list is self-populated with existing owners)
   - **Type**: Select a device type from the dropdown list (Smartphone, Table,
       Laptop, TV, router, ....) or type a new device type
-  - **Vendor**: Automatically updated by Pi.Alert
+  - **Vendor**: Automatically updated by Pi.Alert when empty or unknown
   - **Favorite**: Mark the device as favorite and then it will appears at the
       begining of the device list
   - **Group**: Select a grouper ('Always on', 'Personal', Friends') or type

docs/ICONS.md

@@ -0,0 +1,30 @@
+## Icons overview
+
+Icons are used to visually distinguish devices in the app in most of the device listing tables and the [network tree](/docs/NETWORK_TREE.md). Currently only free [Font Awesome](https://fontawesome.com/search?o=r&m=free) icons (up-to v 6.4.0) are supported (I have an unblockable [sponsorship goal](https://github.com/sponsors/jokob-sk) to add the material design icon pack).  
+
+![Raspberry Pi with a brand icon](/docs/img/ICONS/devices-icons.png)
+
+## ⚙ How to use custom device Icons
+
+You can assign icons individually on each device in the Details tab.
+
+![Raspberry Pi device details](/docs/img/ICONS/device-icon.png)
+
+- You can click into the `Icon` field or click the Pencil (2) icon in the above screenshot to enter any text. Only [free Font Awesome](https://fontawesome.com/search?o=r&m=free) icons in the following format will work:
+
+  1. For any value that is only prefixed with `fa-`, you can enter the value directly, such as `server`, `tv`, `ethernet`. 
+  2. If you want to add another classname, e.g. `fa-brands`, you can enter `brands fa-[fontawesome-icon-name]`, so for `apple` that is using the syntax`fa-brands fa-apple`, you would enter `brands fa-apple`.
+
+- If you want to mass-apply an icon to all devices of the same device type (Field marked (4) in the above screenshot), you can click the copy button (Marked (1) in the above screenshot). A confirmation prompt is displayed. If you proceed, icons of all devices set to the same device type as the current device, will be overwritten with the current device's icon.
+
+- The dropdown (3) contains all icons already used in the app for device icons. You need to navigate away or refresh the page once you add a new icon. 
+
+## 🌟 Pro Font Awesome icons
+
+If you own the premium package of Font Awesome icons you can mount it in your Docker container the following way:
+
+```yaml
+/font-awesome:/home/pi/pialert/front/lib/AdminLTE/bower_components/font-awesome:ro
+```
+
+You can use the full range of Font Awesome icons afterwards. 

docs/NETWORK_TREE.md

@@ -1,8 +1,19 @@
 ## How to setup your Network page
 
-Make sure you have a root device with the MAC `Internet` (No other MAC addresses are currently support as root)
+Make sure you have a root device with the MAC `Internet` (No other MAC addresses are currently supported as the root node).
 
-To setup a device named `rapberrypi` as a `Switch` in our network. 
+## ⚡Quick setup:
+
+* Go to Devices > Device Details. 
+* Find the device(s) you want to use as network devices (network nodes). 
+* Set the Type of such a device to one of the following: AP, Firewall, Gateway, PLC, Powerline, Router, Switch, USB LAN Adapter, USB WIFI Adapter and WLAN.
+* Save and go to Network where the devices you've marked as network devices (by selecting the Type as mentioned above) will show up as tabs.
+* You can now assign the Unassigend devices to the correct network node.
+
+
+## 🔍Detailed example:
+
+In this example you will setup a device named `rapberrypi` as a `Switch` in our network. 
 
 ### 1) Device details page
 

docs/README.md

@@ -0,0 +1,102 @@
+## Documentation overview
+
+In the app hover-over settings or fields/labels or click blue in-app ❔ (question-mark) icons to get to relevant documentation pages.
+
+![In-app help](/docs/img/GENERAL/in-app-help.png)
+
+There is also an in-app Help / FAQ section that should be answering frequently asked questions.
+
+### 📥 Installation
+
+ ⚠ Only tested as a [docker container - follow these instructions here](https://github.com/jokob-sk/Pi.Alert/blob/main/dockerfiles/README.md). 
+ > Check out [leiweibau's fork](https://github.com/leiweibau/Pi.Alert/) if you want to install Pi.Alert on the server directly or original instructions for [pucherot's original code](https://github.com/pucherot/Pi.Alert/)
+
+
+### 📚 Table of contents
+
+#### Popular/Suggested
+
+- [API endpoints details](/docs/API.md)
+- [Plugin system details and how to develop your own](/front/plugins/README.md)
+- [Network tree map configuration](/docs/NETWORK_TREE.md)
+- [Gmail as SMTP server for sending emails](/docs/SMTP_GMAIL.md)
+- [Subnets and vlans configuration for arp-scan](/docs/SUBNETS.md)
+
+#### System Management
+
+- [Manage devices (legacy docs)](/docs/DEVICE_MANAGEMENT.md)
+- [Random MAC/MAC icon meaning (legacy docs)](/docs/RANDOM_MAC.md)
+- [Custom Icons configuration and support](/docs/ICONS.md)
+
+#### Examples
+
+- [N8N webhook example](/docs/WEBHOOK_N8N.md)
+
+#### Misc
+
+- [New Version notifications](/docs/VERSIONS.md)
+- [Version history (legacy)](/docs/VERSIONS_HISTORY.md)
+- [Invalid JSON errors debug help](/docs/DEBUG_INVALID_JSON.md)
+- [Database structure](/docs/DATABASE.md)
+
+Feel free to suggest or submit new docs via a PR. 
+
+## 👨‍💻 Development priorities
+
+Highest to lowest:
+
+* Fixing core functionality bugs not solvable with workarounds
+* New core functionality unlocking other opportunities (e.g.: plugins) 
+* Refactoring enabling faster implementation of future functionality 
+* UI improvements
+
+Design philosophy: Focus on core functionality and leverage existing apps and tools to make PiAlert integratable into other workflows. 
+
+Examples: 
+
+    1. Supporting apprise makes more sense than implementing multiple individual notification gateways
+    2. Implementing regular expressions support across settings for validation makes more sense than validating one setting with a specific expression. 
+
+UI specific requests are low priority as the framework picked by the original developer is not very extensible (and afaik doesn't support components) and has limited mobile support. Also I argue the value proposition is smaller than working on something else.
+
+Feel free to submit PRs if interested. try to **keep the PRs small/on topic** so they are easier to review and approve. 
+
+That being said, I'd reconsider if more people and or recurring sponsors file a request 😉.
+
+## 🙏 Feature requests
+
+Please be as detailed as possible with **workarounds** you considered and why a native feature is the better way. This gives me better context and will make it more likely to be implemented. Ideally a feature request should be in the format "I want to be able to do XYZ so that ZYX. I considered these approaches XYZ".
+
+## ➕ Pull-requests (PRs)
+
+If you submit a PR please:
+
+1. Check that your changes are backward compatible with existing installations and with a blank setup. 
+2. Existing features should always be preserved. 
+3. Keep the PR small, on-topic and don't change code that is not necessary for the PR to work
+4. New features code should ideally be re-usable for different purposes, not be for a very narrow use-case.
+5. New functionality should ideally be implemented via the Plugins system, if possible.
+
+Suggested test cases:
+
+- Blank setup with no DB or config
+- Existing DB / config
+- Sending a notification (e. g. Delete a device and wait for a scan to run) and testing all notification gateways, especially:
+-   Email, Apprise (e.g. via Telegram), webhook (e.g. via Discord), MQTT (e.g. via HomeAssitant)
+- Saving settings
+- Test a couple of plugins
+- Check the Error log for anything unusual
+
+Some additional context:
+
+* Permanent settings/config is stored in the `pialert.conf` file
+* Currently temporary (session?) settings are stored in the `Parameters` DB table as key - value pairs. This table is wiped during a container rebuild/restart and it's values re-initialized from cookies / session data from the browser. 
+
+## 🐛 Submitting an issue or bug
+
+Before submitting a new issue please spend a couple of minutes on research:
+
+* Check [🛑 Common issues](https://github.com/jokob-sk/Pi.Alert/tree/main/dockerfiles#-common-issues) 
+* Check [💡 Closed issues](https://github.com/jokob-sk/Pi.Alert/issues?q=is%3Aissue+is%3Aclosed) if a similar issue was solved in the past.
+
+⚠ Please follow the pre-defined issue template to resolve your issue faster.

docs/SUBNETS.md

@@ -9,7 +9,7 @@ For example, a `/24` mask results in 256 IPs to check, where as a `/16` mask che
 - Run `iwconfig` in your container to find your interface name(s) (e.g.: `eth0`, `eth1`). 
 - Append e.g.: ` -vlan=107` to the interface field (e.g.: `eth0 -vlan=107`) for multiple vlans. More details in this [comment in this issue](https://github.com/jokob-sk/Pi.Alert/issues/170#issuecomment-1419902988)
 
-### Example:
+### 🔍Example:
 
 ![Vlan configuration example](/docs/img/SUBNETS/subnets_vlan.png)
 

docs/VERSIONS.md

@@ -0,0 +1,25 @@
+## Am I running the latest released version?
+
+Since version 23.01.14 PiAlert uses a simple timestamp-based version check to verify if a new version is available. You can check the [current and past releases here](https://github.com/jokob-sk/Pi.Alert/releases), or have a look at what I'm [currently working on](https://github.com/jokob-sk/Pi.Alert/issues/138). 
+
+If you are not on the latest version, the app will notify you, that a new released version is avialable the following way:
+
+### 📧 Via email on a notification event
+
+If any notification occurs and an email is sent, the email will contain a note that a new version is available. See the sample email below:
+
+![Sample email if a new version is available](/docs/img/VERSIONS/new-version-available-email.png)
+
+### 🆕 In the UI
+
+In the UI via a notification Icon and via a custom message in the Maintenance section.
+
+![UI screenshot if a new version is available](/docs/img/VERSIONS/new-version-available-maintenance.png)
+
+For a comparison, this is how the UI looks like if you are on the latest stable image:
+
+![UI screenshot if on latest version](/docs/img/VERSIONS/latest-version-maintenance.png)
+
+## Implementation details
+
+During build a [/home/pi/pialert/front/buildtimestamp.txt](https://github.com/jokob-sk/Pi.Alert/blob/092797e75ccfa8359444ad149e727358ac4da05f/Dockerfile#L44) file is created. The app then periodically checks if a new release is available with a newer timestamp in GitHub's rest-based JSON endpoint (check the `def isNewVersion():` method in `pialert.py` for details).   

front/deviceDetails.php

@@ -194,7 +194,7 @@
                       <div class="form-group">
                         <label class="col-sm-3 control-label">
                           <?= lang('DevDetail_Icon');?> 
-                          <a href="https://fontawesome.com/search?q=laptop&o=r&m=free" target="_blank"> <span><i class="fa fa-fw fa-arrow-up-right-from-square"></i></a><span>
+                          <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/ICONS.md" target="_blank"> <span><i class="fa fa-circle-question"></i></a><span>
                         </label>
                         <div class="col-sm-9">
                           <div class="input-group">

front/devices.php

@@ -225,6 +225,8 @@ function main () {
       // save the columns order in the Devices page 
       tableColumnOrder = numberArrayFromString(data);
 
+
+
       //initialize the table headers in the correct order
       var headersDefaultOrder = [ '<?= lang('Device_TableHead_Name');?>',
                                   '<?= lang('Device_TableHead_Owner');?>',
@@ -308,21 +310,13 @@ function mapIndx(oldIndex)
 function initializeDatatable () {
   for(i = 0; i < tableColumnOrder.length; i++)
   {    
-    // hide this column if not in the tableColumnVisible variable
+    // hide this column if not in the tableColumnVisible variable (we need to keep the MAC address (index 11) for functionality reasons)    
     if(tableColumnVisible.includes(tableColumnOrder[i]) == false)
     {
       tableColumnHide.push(mapIndx(tableColumnOrder[i]));
     }    
   }
-
-  // If the device has a small width (mobile) only show name, ip, and status columns. 
-  if (window.screen.width < 400) {        
-    tableColumnHide = [11,12,13,1,2,4,5,6,7,9];
-  } 
-  // else {  
-  //   // var tableColumnHide = [11, 12, 13];
-  //   tableColumnHide = [11, 12, 13];
-  // };
+  
   var table=
   $('#tableDevices').DataTable({
     'paging'       : true,
@@ -348,7 +342,7 @@ function initializeDatatable () {
 
       // Device Name
       {targets: [mapIndx(0)],
-        'createdCell': function (td, cellData, rowData, row, col) {
+        'createdCell': function (td, cellData, rowData, row, col) {            
             $(td).html ('<b class="anonymizeDev"><a href="deviceDetails.php?mac='+ rowData[mapIndx(11)] +'" class="">'+ cellData +'</a></b>');
       } },
 

front/maintenance.php

@@ -131,8 +131,11 @@ if (isset($_POST['submit']) && submit && isset($_POST['skinselector_set'])) {
               </div>
               <div class="box-body" style="padding-bottom: 5px;">
                 <div class="db_info_table">
-                    <div class="db_info_table_row">
-                        <div class="db_info_table_cell" style="min-width: 140px"><?= lang('Maintenance_version');?></div>
+                    <div class="db_info_table_row">                      
+                        <div class="db_info_table_cell" style="min-width: 140px"><?= lang('Maintenance_version');?>
+                          <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/VERSIONS.md" target="_blank"> <span><i class="fa fa-circle-question"></i></a><span>
+
+                        </div>
                         <div class="db_info_table_cell">
                         <div class="version" id="version" data-build-time="<?php echo file_get_contents( "buildtimestamp.txt");?>"><?php echo '<span id="new-version-text" class="myhidden">' .lang('Maintenance_new_version').'</span>'.'<span id="current-version-text" class="myhidden">' .lang('Maintenance_current_version').'</span>';?></div>
                         </div>
@@ -335,7 +338,7 @@ if (isset($_POST['submit']) && submit && isset($_POST['skinselector_set'])) {
                     <div class="db_info_table">
                         <div class="log-area">
                             <div class="row logs-row">
-                              <textarea id="pialert_log" class="logs" cols="70" rows="10" wrap='off' readonly ><?php echo file_get_contents( "./log/pialert.log" ); ?>
+                              <textarea id="pialert_log" class="logs" cols="70" rows="10" wrap='off' readonly ><?php echo file_get_contents( "./log/pialert.log", false, null, -200000); ?>
                               </textarea>
                             </div>
                             <div class="row logs-row" >
@@ -364,7 +367,24 @@ if (isset($_POST['submit']) && submit && isset($_POST['skinselector_set'])) {
                                 </div>
                               </div>
                             </div>
-                        </div>                           
+                        </div> 
+                        <div class="log-area">
+                            <div class="row logs-row">
+                              <textarea id="pialert_php_log" class="logs" cols="70" rows="10" wrap='off' readonly><?php echo file_get_contents( "./log/pialert.php_errors.log" ); ?>
+                              </textarea>
+                            </div>
+                            <div class="row logs-row" >                            
+                              <div>
+                                <div class="log-file">pialert.php_errors.log<div class="logs-size"><?php echo number_format((filesize("./log/pialert.php_errors.log") / 1000000),2,",",".") . ' MB';?> 
+                                <span class="span-padding"><a href="./log/pialert.php_errors.log"><i class="fa fa-download"></i> </a></span>
+                              </div></div>
+                                <div class="log-purge">
+                                  <button class="btn btn-primary" onclick="logManage('pialert.php_errors.log','cleanLog')"><?= lang('Gen_Purge');?></button>
+                                </div>
+                              </div>
+                            </div>
+                        </div>                         
+                        
                          <div class="log-area">
                              
                              <div class="row logs-row">
@@ -702,12 +722,16 @@ function performLogManage() {
 // --------------------------------------------------------
 function scrollDown()
 {
-  var areaIDs = ['pialert_log', 'pialert_front_log', 'IP_changes_log', 'stdout_log', 'stderr_log', 'pialert_pholus_log',  'pialert_pholus_lastrun_log'];
+  var areaIDs = ['pialert_log', 'pialert_front_log', 'IP_changes_log', 'stdout_log', 'stderr_log', 'pialert_pholus_log',  'pialert_pholus_lastrun_log', 'pialert_php_log'];
   
   for (let i = 0; i < areaIDs.length; i++) {
 
     var tempArea = $('#' + areaIDs[i]);
-    $(tempArea[0]).scrollTop(tempArea[0].scrollHeight);
+    
+    if (tempArea.length > 0)
+    {
+      $(tempArea[0]).scrollTop(tempArea[0].scrollHeight);
+    }
 
   }
 }
@@ -715,8 +739,8 @@ function scrollDown()
 // --------------------------------------------------------
 // Manage displayed columns
 // --------------------------------------------------------
-colDefaultOrder = ['0','1','2','3','4','5','6','7','8','9','10','12','13','14','15','16','17'];
-colDefaultOrderTxt = '[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,17]';
+colDefaultOrder = ['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17'];
+colDefaultOrderTxt = '[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]';
 
 
 function saveSelectedColumns () { 
@@ -724,9 +748,7 @@ function saveSelectedColumns () {
     // save full order of all columns to simplify mapping later on
     
     colDisplayed = $('#columnsSelect').val();
-
     
-
     colNewOrder = colDisplayed;
 
     // append the remaining columns in the previous order

front/php/server/devices.php

@@ -570,72 +570,6 @@ function getDevicesTotals() {
 }
 
 
-//------------------------------------------------------------------------------
-//  Query the List of devices in a determined Status
-//------------------------------------------------------------------------------
-// function getDevicesListForNetworkTree() {
-//   global $db;
-
-//   $sql = 'SELECT *, CASE
-//               WHEN t1.dev_AlertDeviceDown=1 AND t1.dev_PresentLastScan=0 THEN "Down"
-//               WHEN t1.dev_NewDevice=1 THEN "New"
-//               WHEN t1.dev_PresentLastScan=1 THEN "On-line"
-//               ELSE "Off-line"  END AS dev_Status
-//               FROM (Devices ) t1
-//           LEFT JOIN
-//                       (
-//                             SELECT *,
-//                                   count() as connected_devices 
-//                             FROM Devices b 
-//                             WHERE b.dev_Network_Node_MAC_ADDR NOT NULL group by b.dev_Network_Node_MAC_ADDR
-//                       ) t2
-//                       ON (t1.dev_MAC = t2.dev_MAC); ';
-
-//   $result = $db->query($sql);
-
-//   // arrays of rows
-//   $tableData = array();
-//   while ($row = $result -> fetchArray (SQLITE3_ASSOC)) {
-
-//   $defaultOrder = array ($row['dev_Name'],
-//                     $row['dev_Owner'],
-//                     handleNull($row['dev_DeviceType']),
-//                     handleNull($row['dev_Icon'], "laptop"),
-//                     $row['dev_Favorite'],
-//                     $row['dev_Group'],
-//                     formatDate ($row['dev_FirstConnection']),
-//                     formatDate ($row['dev_LastConnection']),
-//                     $row['dev_LastIP'],
-//                     ( in_array($row['dev_MAC'][1], array("2","6","A","E","a","e")) ? 1 : 0),
-//                     $row['dev_Status'],
-//                     $row['dev_MAC'], // MAC (hidden)
-//                     formatIPlong ($row['dev_LastIP']), // IP orderable
-//                     $row['rowid'], // Rowid (hidden)      
-//                     handleNull($row['dev_Network_Node_MAC_ADDR']), // 
-//                     handleNull($row['connected_devices']) // 
-//                     );
-
-//   $newOrder = array();
-
-//   // reorder columns based on user settings
-//   for($index = 0; $index  < count($columnOrderMapping); $index++)
-//   {
-//   array_push($newOrder, $defaultOrder[$columnOrderMapping[$index][2]]);      
-//   }
-
-//   $tableData['data'][] = $newOrder;
-//   }
-
-//   // Control no rows
-//   if (empty($tableData['data'])) {
-//   $tableData['data'] = '';
-//   }
-
-//   // Return json
-//   echo (json_encode ($tableData));
-
-// }
-
 //------------------------------------------------------------------------------
 //  Query the List of devices in a determined Status
 //------------------------------------------------------------------------------
@@ -649,7 +583,7 @@ function getDevicesList() {
     $forceDefaultOrder = TRUE;
   }
 
-  // This object is used to map from the old order ( second parameter, first number) to the 3rd parameter (Second number (here initialized to -1))
+  // This object is used to map from the old order ( second parameter, first number) to the new mapping, that is represented by the 3rd parameter (Second number)
   $columnOrderMapping = array(
     array("dev_Name", 0, 0),               
     array("dev_Owner", 1, 1),              
@@ -684,9 +618,11 @@ function getDevicesList() {
       $orderedColumns = createArray($row[0]);
 
       // init ordered columns    
-      for($i = 0; $i < count($orderedColumns); $i++) {           
-        $columnOrderMapping[$i][2] = $orderedColumns[$i];      
+      for($i = 0; $i < count($orderedColumns); $i++) { 
+
+        $columnOrderMapping[$i][2] = $orderedColumns[$i];        
       }
+
     }
   }
 

front/php/templates/footer.php

@@ -12,9 +12,38 @@
     <!-- Default to the left -->
 
     <!-- &copy; 2020 Puche -->
-    <?php
-      echo '<span style="display:inline-block; transform: rotate(180deg)">&copy;</span> 2020 Puche (2022+ <a href="mailto:jokob@duck.com?subject=PiAlert">jokob-sk</a>)';
-    ?>
+    <span style="display:inline-block; transform: rotate(180deg)">&copy;</span>
+
+       
+    
+       2020 Puche (2022+ <a href="mailto:jokob@duck.com?subject=PiAlert">jokob-sk</a>) | <b>Built on: </b> 
+       
+       <?php 
+        
+        echo date("Y-m-d", ((int)file_get_contents( "buildtimestamp.txt")));        
+       
+       ?> 
+       
+       | <b> Version: </b>
+       
+       <?php      
+
+        $filename = "/.VERSION";
+       
+        if(file_exists($filename))
+        {
+          echo file_get_contents($filename);
+        }
+        else{
+          echo "File not found";
+        }               
+       
+       ?>
+        |
+      <a href="https://github.com/jokob-sk/Pi.Alert/tree/main/docs" target="_blank"> 
+        <span>Docs <i class="fa fa-circle-question"></i>
+      </a><span>
+    
     <!-- To the right -->
     <div class="pull-right no-hidden-xs">
 

front/php/templates/graph.php

@@ -8,15 +8,43 @@ $Pia_Graph_Device_Online = array();
 $Pia_Graph_Device_Down = array();
 $Pia_Graph_Device_Arch = array();
 
+$statusesToShow = "'online', 'offline', 'archived'";
+
+$statQuery = $db->query("SELECT * FROM Settings WHERE Code_Name = 'UI_PRESENCE'");
+
+while($r = $statQuery->fetchArray(SQLITE3_ASSOC))
+{
+  $statusesToShow =  $r['Value'];
+}
+
 $results = $db->query('SELECT * FROM Online_History ORDER BY Scan_Date DESC LIMIT 144');
-while ($row = $results->fetchArray()) {
-   $time_raw = explode(' ', $row['Scan_Date']);
-   $time = explode(':', $time_raw[1]);
-   array_push($Pia_Graph_Device_Time, $time[0].':'.$time[1]);
-   array_push($Pia_Graph_Device_Down, $row['Down_Devices']);
-   array_push($Pia_Graph_Device_All, $row['All_Devices']);
-   array_push($Pia_Graph_Device_Online, $row['Online_Devices']);
-   array_push($Pia_Graph_Device_Arch, $row['Archived_Devices']);
+
+while ($row = $results->fetchArray()) 
+{
+    $time_raw = explode(' ', $row['Scan_Date']);
+    $time = explode(':', $time_raw[1]);
+    array_push($Pia_Graph_Device_Time, $time[0].':'.$time[1]);
+    
+    //  Offline
+    if(strpos($statusesToShow, 'offline') !== false) 
+    {
+      array_push($Pia_Graph_Device_Down, $row['Down_Devices']);
+    }
+
+    //  All
+    array_push($Pia_Graph_Device_All, $row['All_Devices']);
+
+    // Online
+    if(strpos($statusesToShow, 'online') !== false)
+    {
+      array_push($Pia_Graph_Device_Online, $row['Online_Devices']);
+    }
+    
+    // Archived
+    if(strpos($statusesToShow, 'archived') !== false)
+    {   
+      array_push($Pia_Graph_Device_Arch, $row['Archived_Devices']);
+    }
 }
 function pia_graph_devices_data($Pia_Graph_Array) {
   $Pia_Graph_Array_rev = array_reverse($Pia_Graph_Array);

front/php/templates/language/en_us.php

@@ -116,6 +116,7 @@ $lang['en_us'] = array(
 'Presence_CalHead_quarter' => 'quarter',
 'Presence_CalHead_month' => 'month',
 'Presence_CalHead_week' => 'week',
+'Presence_CalHead_day' => 'day',
 
 //////////////////////////////////////////////////////////////////
 // Events Page
@@ -214,7 +215,7 @@ $lang['en_us'] = array(
 'DevDetail_button_Reset' => 'Reset Changes',
 'DevDetail_button_Save' => 'Save',
 'DevDetail_button_OverwriteIcons' => 'Overwrite Icons',
-'DevDetail_button_OverwriteIcons_Tooltip' => 'Overwrite icons of all devices with the same type',
+'DevDetail_button_OverwriteIcons_Tooltip' => 'Overwrite icons of all devices with the same device type',
 'DevDetail_button_OverwriteIcons_Warning' => 'Are you sure you want to overwrite all icons of all devices with the same device type as the current device type?',
 'DevDetail_SessionTable_Order' => 'Order',
 'DevDetail_SessionTable_Connection' => 'Connection',
@@ -515,7 +516,7 @@ $lang['en_us'] = array(
 'ENABLE_ARPSCAN_description' => 'Arp-scan is a command-line tool that uses the ARP protocol to discover and fingerprint IP hosts on the local network. An alternative to ARP scan is to enable the <a onclick="toggleAllSettings()" href="#PIHOLE_ACTIVE"><code>PIHOLE_ACTIVE</code>PiHole integration settings</a>.',
 'SCAN_SUBNETS_name' => 'Subnets to scan',
 'SCAN_SUBNETS_description' => '
-The arp-scan time itself depends on the number of IP addresses to check so set this up carefully with the appropriate network mask and interface. Check the <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SUBNETS.md" target="_blank">subnets documentation</a> for details.
+The arp-scan time itself depends on the number of IP addresses to check so set this up carefully with the appropriate network mask and interface. Check the <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SUBNETS.md" target="_blank">subnets documentation</a> for help on setting up VLANs, what VLANs are supported, or how to figure out the network mask and your interface.
 ',
 'LOG_LEVEL_name' => 'Print additional logging',
 'LOG_LEVEL_description' => 'This setting will enable more verbose logging. Useful for debugging events writing into the database.',
@@ -528,7 +529,7 @@ The arp-scan time itself depends on the number of IP addresses to check so set t
 'PIALERT_WEB_PASSWORD_name' => 'Login password',
 'PIALERT_WEB_PASSWORD_description' => 'The default password is <code>123456</code>. To change the password run <code>/home/pi/pialert/back/pialert-cli</code> in the container',
 'INCLUDED_SECTIONS_name' => 'Notify on',
-'INCLUDED_SECTIONS_description' => 'Specifies which events trigger notifications. Remove the event type(s) you don\'t want to get notified on. This setting overrides device-specific settings in the UI. (<code>CTRL + Click</code> to select / deselect).',
+'INCLUDED_SECTIONS_description' => 'Specifies which events trigger notifications. Remove the event type(s) you don\'t want to get notified on. This setting overrides device-specific settings in the UI. (<code>CTRL + Click</code> to select/deselect).',
 'SCAN_CYCLE_MINUTES_name' => 'Scan cycle delay',
 'SCAN_CYCLE_MINUTES_description' => 'The delay between scans in minutes. If using arp-scan, the scan time itself depends on the number of IP addresses to check. This is influenced by the network mask set in the <a href="#SCAN_SUBNETS"><code>SCAN_SUBNETS</code> setting</a> at the top. Every IP takes a couple seconds to scan.',
 'DAYS_TO_KEEP_EVENTS_name' => 'Delete events older than',
@@ -539,6 +540,8 @@ The arp-scan time itself depends on the number of IP addresses to check so set t
 'DIG_GET_IP_ARG_description' => 'Change the <a href="https://linux.die.net/man/1/dig" target="_blank">dig utility</a> arguments if you have issues resolving your Internet IP. Arguments are added at the end of the following command: <code>dig +short </code>.',
 'UI_LANG_name' => 'UI Language',
 'UI_LANG_description' => 'Select the preferred UI language.',
+'UI_PRESENCE_name' => 'Show in presence chart',
+'UI_PRESENCE_description' => 'Select what statuses should be shown in the <b>Device presence over time</b>  chart in the <a href="/devices.php" target="_blank">Devices</a> page. (<code>CTRL + Click</code> to select/deselect)',
 
 //Email
 'Email_display_name' => 'Email',

front/php/templates/language/es_es.php

@@ -5,6 +5,7 @@ $lang['es_es'] = array(
 //////////////////////////////////////////////////////////////////
 // About - Update by @TeroRERO 07ago2022
 //////////////////////////////////////////////////////////////////
+
 'About_Title' => 'Guadián de Red <br>(Código Abierto)',
 'About_Design' => 'Diseñado para:',
 'About_Exit' => 'Salir',
@@ -12,20 +13,31 @@ $lang['es_es'] = array(
 //////////////////////////////////////////////////////////////////
 // General - Update by @TeroRERO 01ago2022
 //////////////////////////////////////////////////////////////////
+
 'Gen_Delete' => 'Eliminar',
+'Gen_DeleteAll' => 'Eliminar todo',								
 'Gen_Cancel' => 'Cancelar',
-'Gen_Okay' => 'Ok',
+'Gen_Okay' => 'Aceptar',
+'Gen_Save' => 'Guardar',
+'Gen_Saved' => 'Guardado',
+'Gen_Run' => 'Ejecutar',
+'Gen_Action' => 'Acción',
 'Gen_Purge' => 'Purgar',
 'Gen_Backup' => 'Ejecutar copia de seguridad',
 'Gen_Restore' => 'Ejecutar restauración',
 'Gen_Switch' => 'Cambiar',
 'Gen_AreYouSure' => '¿Estás seguro de',
+'Gen_Upd' => 'Actualizado correctamente',
+'Gen_Upd_Fail' => 'Fallo al actualizar',
+'Gen_Help' => 'Ayuda',
+'Gen_DataUpdatedUITakesTime' => 'Correcto - La interfaz puede tardar en actualizarse si se está ejecutando un escaneo.',
+'Gen_LockedDB' => 'Fallo - La base de datos puede estar bloqueada - Pulsa F1 -> Ajustes de desarrolladores -> Consola o prueba más tarde.',		
 
 //////////////////////////////////////////////////////////////////
 // Login Page - Update by @TeroRERO 03ago2022
 //////////////////////////////////////////////////////////////////
-
 // TeroRERO Off 'Login_Box' => 'Inicie su sesión',
+
 'Login_Box' => 'Ingrese su contraseña',
 'Login_Remember' => 'Recordar',
 'Login_Remember_small' => '(válido por 7 días)',
@@ -38,6 +50,7 @@ $lang['es_es'] = array(
 'Login_Toggle_Info' => 'Información sobre la contraseña',
 'Login_Toggle_Info_headline' => 'Información sobre la contraseña',
 'Login_Toggle_Alert_headline' => 'Alerta de Contraseña!',
+'Login_Default_PWD' => 'La contraseña por defecto "123456" sigue activa.',																	
 
 //////////////////////////////////////////////////////////////////
 // Device Page - Update by @TeroRERO 03ago2022
@@ -46,10 +59,11 @@ $lang['es_es'] = array(
 'Navigation_Devices' => 'Dispositivos',
 'Navigation_Presence' => 'Historial',
 'Navigation_Events' => 'Eventos',
+'Navigation_Network' => 'Red',
+'Navigation_Plugins' => 'Plugins',
 'Navigation_Maintenance' => 'Mantenimiento',
 'Navigation_Settings' => 'Configuración',
-'Navigation_Network' => 'Red',
-'Navigation_HelpFAQ' => 'Ayuda / FAQ',
+'Navigation_HelpFAQ' => 'Ayuda / Preguntas frecuentes',				
 'Device_Title' => 'Dispositivos',
 'Device_Shortcut_AllDevices' => 'Todos',
 'Device_Shortcut_Connected' => 'Conectado(s)',
@@ -62,14 +76,21 @@ $lang['es_es'] = array(
 'Device_TableHead_Name' => 'Nombre',
 'Device_TableHead_Owner' => 'Propietario',
 'Device_TableHead_Type' => 'Tipo',
+'Device_TableHead_Icon' => 'Icon',
+'Device_TableHead_RowID' => 'Row ID',
+'Device_TableHead_Rowid' => 'Row ID',
+'Device_TableHead_Parent_MAC' => 'Nodo principal de la MAC',
+'Device_TableHead_Connected_Devices' => 'Dispositivos conectados',
+'Device_TableHead_Location' => 'Ubicación',
+'Device_TableHead_Vendor' => 'Fabricante',									 
 'Device_TableHead_Favorite' => 'Favorito',
 'Device_TableHead_Group' => 'Grupo',
 'Device_TableHead_FirstSession' => '1ra. sesión',
 'Device_TableHead_LastSession' => 'Última sesión',
 'Device_TableHead_LastIP' => 'Última IP',
 'Device_TableHead_MAC' => 'MAC',
+'Device_TableHead_MAC_full' => 'MAC completa',
 'Device_TableHead_LastIPOrder' => 'Última orden de IP',
-'Device_TableHead_Rowid' => 'Rowid',
 'Device_TableHead_Status' => 'Estado',
 'Device_Searchbox' => 'Búsqueda',
 'Device_Tablelenght' => 'Mostrar _MENU_ entradas',
@@ -153,17 +174,26 @@ $lang['es_es'] = array(
 'DevDetail_Tab_Sessions' => 'Sesiones',
 'DevDetail_Tab_Presence' => 'Historial',
 'DevDetail_Tab_Events' => 'Eventos',
+'DevDetail_Tab_Pholus' => '<i class="fa fa-search"></i> Pholus',
+'DevDetail_Tab_PholusEmpty' => 'No se ha encontrado nada para este dispositivo con Pholus.',
+'DevDetail_Tab_NmapTableHeader' => 'Resultados del escaneo programado',
+'DevDetail_Tab_NmapTableText' => 'Establece la programación en los <a href="/settings.php#NMAP_ACTIVE">Ajustes</a>',
+'DevDetail_Tab_NmapEmpty' => 'Ningún puerto detectado en este dispositivo con Nmap.',
 'DevDetail_MainInfo_Title' => 'Información principal',
 'DevDetail_MainInfo_mac' => 'MAC',
 'DevDetail_MainInfo_Name' => 'Nombre',
 'DevDetail_MainInfo_Owner' => 'Propietario',
 'DevDetail_MainInfo_Type' => 'Tipo',
+'DevDetail_Icon' => 'Icono',
+'DevDetail_Icon_Descr' => 'Enter a font awesome icon name without the fa- prefix or with complete class, e.g.: fa fa-brands fa-apple.',
 'DevDetail_MainInfo_Vendor' => 'Proveedor',
 'DevDetail_MainInfo_Favorite' => 'Favorito',
 'DevDetail_MainInfo_Group' => 'Grupo',
 'DevDetail_MainInfo_Location' => 'Ubicación',
 'DevDetail_MainInfo_Comments' => 'Comentario',
 'DevDetail_MainInfo_Network' => 'Hardware de Red (ID)',
+'DevDetail_MainInfo_Network' => '<i class="fa fa-server"></i> Nodo (MAC)',
+'DevDetail_GoToNetworkNode' => 'Navegar a la página de Internet del nodo seleccionado.',
 'DevDetail_MainInfo_Network_Port' => 'Puerto de Red HW',
 'DevDetail_SessionInfo_Title' => 'Información de sesión',
 'DevDetail_SessionInfo_Status' => 'Estado',
@@ -183,7 +213,12 @@ $lang['es_es'] = array(
 'DevDetail_EveandAl_ScanCycle_z' => 'No Escanear Dispositivo',
 'DevDetail_button_Delete' => 'Eliminar dispositivo',
 'DevDetail_button_Reset' => 'Restablecer cambios',
+'DevDetail_button_DeleteEvents_Warning' => '¿Desea eliminar todos los eventos de este dispositivo?<br><br>(se eliminarán el <b>Historial de eventos</b> y las <b>Sesiones</b>, y puede ayudar en el caso de notificaciones constantes)',
+'DevDetail_button_Reset' => 'Restablecer cambios',
 'DevDetail_button_Save' => 'Guardar',
+'DevDetail_button_OverwriteIcons' => 'Sobreescribir iconos',
+'DevDetail_button_OverwriteIcons_Tooltip' => 'Sobreescribir los iconos de todos los dispositivos con el mismo tipo',
+'DevDetail_button_OverwriteIcons_Warning' => '¿Sobreescribir todos los iconos de todos los dispositivos con el mismo tipo que el dispositivo actual?',
 'DevDetail_SessionTable_Order' => 'Ordenar',
 'DevDetail_SessionTable_Connection' => 'Conexión',
 'DevDetail_SessionTable_Disconnection' => 'Desconexión',
@@ -199,12 +234,27 @@ $lang['es_es'] = array(
 'DevDetail_Nmap_buttonDetail_text' => 'Escaneo detallado: escaneo predeterminado con detección de sistema operativo habilitado, detección de versiones, escaneo de script y traceroute (hasta 30 segundos o más)',
 'DevDetail_Nmap_buttonSkipDiscovery' => 'Omitir detección de host',
 'DevDetail_Nmap_buttonSkipDiscovery_text' => 'Omitir detección de host (-Pn opción): Escaneo predeterminado sin detección de host',
+'DevDetail_Nmap_resultsLink' => 'Puedes abandonar esta página después de empezar un escaneo. Los resultados también estarán disponibles en el archivo <code>pialert_front.log</code>.',
+'BackDevDetail_Actions_Title_Run' => 'Ejecutar acción',
+'BackDevDetail_Actions_Not_Registered' => 'Acción no registrada: ',
+'BackDevDetail_Actions_Ask_Run' => '¿Desea ejecutar la acción?',
+'BackDevDetail_Tools_WOL_okay' => 'El comando se ha ejecutado correctamente.',
+'BackDevDetail_Tools_WOL_error' => 'Ha ocurrido un error al ejectuar el comando.',
+'DevDetail_Tools_WOL_noti' => 'Wake-on-LAN',
+'DevDetail_Tools_WOL_noti_text' => 'El comando de Wake-on-LAN en enviado a la dirección de escucha. Si el dispositivo no está en la misma subred/vlan que Pi.Alert, el dispositivo no responderá.',
+'DevDetail_Tools_WOL' => 'Enviar comando WOL a ',
+'DevDetail_WOL_Title' => '<i class="fa fa-power-off"></i> Wake-on-LAN',
+'DevDetail_Run_Actions_Title' => '<i class="fa fa-play"></i> Ejecutar acción en el dispositivo',
+'DevDetail_Run_Actions_Tooltip' => 'Ejecutar la acción del desplegable sobre el dispositivo actual.',
 
 //////////////////////////////////////////////////////////////////
 // Maintenance Page - Update by @TeroRERO 07ago2022
 //////////////////////////////////////////////////////////////////
 
 'Maintenance_Title' => 'Herramientas de mantenimiento',
+'Maintenance_version' => 'Actualizaciones de la aplicación',
+'Maintenance_new_version' => '🆕 Una nueva versión está disponible. Comprueba las <a href="https://github.com/jokob-sk/Pi.Alert/releases" target="_blank">notas de lanzamiento</a>.',
+'Maintenance_current_version' => 'No hay actualizaciones disponibles. Comprueba en que <a href="https://github.com/jokob-sk/Pi.Alert/issues/138" target="_blank">se está trabajando</a>.',
 'Maintenance_database_path' => 'Ruta de la base de datos:',
 'Maintenance_database_size' => 'Tamaño de base de datos:',
 'Maintenance_database_lastmod' => 'Última modificación:',
@@ -214,10 +264,10 @@ $lang['es_es'] = array(
 'Maintenance_arp_status' => 'Estado de escaneo:',
 'Maintenance_arp_status_off' => 'está actualmente deshabilitado',
 'Maintenance_arp_status_on' => 'escaneo(s) actualmente en ejecución',
-'Maintenance_themeselector_lable' => 'Seleccionar Skin',
-'Maintenance_themeselector_empty' => 'Elija un Skin',
-'Maintenance_themeselector_text' => 'El cambio tiene lugar en el lado del servidor, por lo que afecta todos los dispositivos en uso.',
+'Maintenance_themeselector_lable' => 'Seleccionar tema',
+'Maintenance_themeselector_empty' => 'Elige un tema',
 'Maintenance_themeselector_apply' => 'Aplicar',
+'Maintenance_themeselector_text' => 'El cambio tiene lugar en el lado del servidor, por lo que afecta todos los dispositivos en uso.',
 'Maintenance_lang_selector_lable' => 'Seleccione su idioma',
 'Maintenance_lang_selector_empty' => 'Elija un idioma',
 'Maintenance_lang_en_us' => 'English (US)',
@@ -226,10 +276,15 @@ $lang['es_es'] = array(
 'Maintenance_lang_selector_text' => 'El cambio tiene lugar en el lado del servidor, por lo que afecta todos los dispositivos en uso.',
 'Maintenance_lang_selector_apply' => 'Aplicar',
 'Maintenance_Tools_Tab_Settings' => 'Ajustes',
-'Maintenance_Tools_Tab_Tools' => 'Tools',
+'Maintenance_Tools_Tab_UISettings' => 'Ajustes de interfaz',													
+'Maintenance_Tools_Tab_Tools' => 'Herramientas',
 'Maintenance_Tools_Tab_BackupRestore' => 'Respaldo / Restaurar',
-'Maintenance_Tools_Tab_Logging' => 'Logs',
+'Maintenance_Tools_Tab_Logging' => 'Registros',
+'Maintenance_Tool_displayed_columns_text' => 'Cambia la visibilidad y el orden de las columnas en la página <a href="devices.php"><b> <i class="fa fa-laptop"></i> Dispositivos</b></a> . (La función de coger y arrastrar funciona un poco mal, pero funciona. (Se intentó arreglar <a href="https://github.com/jokob-sk/Pi.Alert/commit/94b32f0f7332879f5a7d2af05dafa2e5d5cfa5da">como por 3 horas</a> - se agradecerían PRs para arreglarlo)).',
+'Maintenance_Tool_order_columns_text' => '',
 'Maintenance_Tool_darkmode' => 'Cambiar Modo (Dark/Light)',
+'Maintenance_Tool_drag_me' => 'Coger para rearrastrar columnas.',
+'Maintenance_Tool_check_visible' => 'Desactivar para ocultar columna.',
 'Maintenance_Tool_darkmode_text' => 'Alternar entre el modo oscuro y el modo de luz. Si el interruptor no funciona correctamente, intente borrar el caché del navegador. El cambio tiene lugar en el lado del servidor, por lo que afecta todos los dispositivos en uso.',
 'Maintenance_Tool_darkmode_noti' => 'Cambiar Modo',
 'Maintenance_Tool_darkmode_noti_text' => 'Después del cambio de tema, la página intenta volver a cargar para activar el cambio. Si es necesario, el caché debe ser eliminado.',
@@ -284,8 +339,6 @@ $lang['es_es'] = array(
 'Maintenance_Tool_ImportCSV_text' => 'Antes de usar esta función, haga una copia de seguridad. Importe un archivo CSV (valor separado por comas) que contiene la lista de dispositivos, incluidas las relaciones de red entre nodos de red y dispositivos conectados. Para hacer eso, coloque el archivo CSV llamado <b> devices.csv </b> en su carpeta <b>/config </b>.',
 'Maintenance_Tool_ImportCSV_noti' => 'Importación CSV',
 'Maintenance_Tool_ImportCSV_noti_text' => '¿Está seguro de que quiere importar el archivo CSV? Esto sobrescribirá completamente los dispositivos de su base de datos.',
-'Maintenance_Github_package_a' => 'La última versión de Pi.Alert (Fork leiweibau) se publicó en ',
-'Maintenance_Github_package_b' => '',
 
 //////////////////////////////////////////////////////////////////
 // Maintenance Page - Update by @TeroRERO 25jul2022
@@ -312,6 +365,10 @@ $lang['es_es'] = array(
 'BackDevices_DBTools_Upgrade' => 'Base de datos actualizada correctamente',
 'BackDevices_DBTools_UpgradeError' => 'Falló la actualización de la base de datos',
 'BackDevices_DBTools_Purge' => 'Las copias de seguridad más antiguas fueron eliminadas',
+'BackDevices_DBTools_ImportCSV' => 'Los dispositivos del archivo CSV han sido importados correctamente.',
+'BackDevices_DBTools_ImportCSVError' => 'El archivo CSV no pudo ser importado. Asegúrate de que el formato es correcto.',
+'BackDevices_DBTools_ImportCSVMissing' => 'El archivo CSV no se pudo encontrar en <b>/config/devices.csv.</b>',
+'BackDevices_Device_UpdDevError' => 'Fallo al actualizar dispositivos, pruebe de nuevo más tarde. La base de datos probablemente esté bloqueada por una tarea en curso.',
 
 //////////////////////////////////////////////////////////////////
 // Network Page - Update by @TeroRERO 01ago2022
@@ -320,6 +377,8 @@ $lang['es_es'] = array(
 'Network_Title' => 'Descripción general de la red',
 'Network_ManageDevices' => 'Administrar dispositivos',
 'Network_ManageAdd' => 'Añadir dispositivo',
+'Network_ManageAssign' => 'Asignar',
+'Network_ManageUnassign' => 'Desasignar',									   
 'Network_ManageEdit' => 'Actualizar dispositivo',
 'Network_ManageDel' => 'Eliminar dispositivo',
 'Network_ManageAdd_Name' => 'Nombre del dispositivo',
@@ -344,6 +403,14 @@ $lang['es_es'] = array(
 'Network_Table_State' => 'Estado',
 'Network_Table_Hostname' => 'Nombre de host',
 'Network_Table_IP' => 'Dirección IP',
+'Network_UnassignedDevices' => 'Dispositivos sin asignar',
+'Network_Assign' => 'Conectar al nodo de <i class="fa fa-server"></i> red',
+'Network_Connected' => 'Dispositivos conectados',
+'Network_ManageLeaf' => 'Gestionar asignación',
+'Network_Node' => 'Nodo de red',
+'Network_Node_Name' => 'Nombre de nodo',
+'Network_Parent' => 'Dispositivo primario de la red',
+'Network_NoAssignedDevices' => 'Este nodo de red no tiene asignado ningún dispositivo (nodo externo). Asigna uno de la lista o ve a la pestaña <b><i class="fa fa-info-circle"></i> Detalles</b> de cualquier dispositivo en la página<a href="devices.php"><b> <i class="fa fa-laptop"></i> Dispositivos</b></a>, y asígnalo a un <b><i class="fa fa-server"></i> Nodo (MAC)</b>  <b><i class="fa fa-ethernet"></i> Puerto</b> de la red ahí.',
 
 //////////////////////////////////////////////////////////////////
 // Help Page - Update by @TeroRERO 07ago2022
@@ -369,13 +436,13 @@ $lang['es_es'] = array(
                 							 chmod -R 770 ~/pialert/db
               								 </span><br>
               								 Si la base de datos sigue siendo de sólo lectura, intente reinstalar o restaurar una copia de seguridad de la base de datos desde la página de mantenimiento.',
-'HelpFAQ_Cat_General_102docker_head' => '(🐳 Docker only) Database issues (AJAX errors, read-only, not found)',
-'HelpFAQ_Cat_General_102docker_text' => 'Double-check you\'ve followed the <a href="https://github.com/jokob-sk/Pi.Alert/tree/main/dockerfiles">dockerfile readme (most up-to-date info)</a>. <br/> <br/> <ul data-sourcepos="49:4-52:146" dir="auto">
-												<li data-sourcepos="49:4-49:106">Download the <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/db/pialert.db">original DB from GitHub</a>.</li>
-												<li data-sourcepos="50:4-50:195">Map the <code>pialert.db</code> file (<g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠</g-emoji> not folder) from above to <code>/home/pi/pialert/db/pialert.db</code> (see <a href="https://github.com/jokob-sk/Pi.Alert/tree/main/dockerfiles#-examples">Examples</a> for details).</li>
-												<li data-sourcepos="51:4-51:161">If facing issues (AJAX errors, can\'t write to DB, etc,) make sure permissions are set correctly, alternatively check the logs under <code>/home/pi/pialert/front/log</code>.</li>
-												<li data-sourcepos="52:4-52:146">To solve permission issues you can also try to create a DB backup and then run a DB Restore via the <strong>Maintenance &gt; Backup/Restore</strong> section.</li>
-												<li data-sourcepos="53:4-53:228">If the database is in read-only mode you can solve this by setting the owner and group by executing the following command on the host system: <code>docker exec pialert chown -R www-data:www-data /home/pi/pialert/db/pialert.db</code>.</li>
+'HelpFAQ_Cat_General_102docker_head' => '(🐳 Solo Docker) Problemas con la base de datos (errores de AJAX, solo lectura, no encontrado)',
+'HelpFAQ_Cat_General_102docker_text' => 'Comprueba que has seguido las instrucciones del <a href="https://github.com/jokob-sk/Pi.Alert/tree/main/dockerfiles">dockerfile (la información más actualizada)</a>. <br/> <br/> <ul data-sourcepos="49:4-52:146" dir="auto">
+												<li data-sourcepos="49:4-49:106">Descarga la <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/db/pialert.db">base de datos original desde GitHub</a>.</li>
+												<li data-sourcepos="50:4-50:195">Mapea el archivo <code>pialert.db</code> (<g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠</g-emoji> no carpeta) de arriba a <code>/home/pi/pialert/db/pialert.db</code> (puedes comprobar los <a href="https://github.com/jokob-sk/Pi.Alert/tree/main/dockerfiles#-examples">ejemplos</a> para más detalles).</li>
+												<li data-sourcepos="51:4-51:161">Si aparecen problemas (errores de AJAX, no se puede escribir a la base de datos, etc,) asegúrate que los permisos están establecidos correctamente. También puedes comprobar los registros en <code>/home/pi/pialert/front/log</code>.</li>
+												<li data-sourcepos="52:4-52:146">Para arreglar los problemas de los permisos, puedes probar a crear una copia de seguridad de la base de datos y después restaurarla desde la sección <strong>Mantenimiento &gt; Copia de seguridad/Restaurar</strong>.</li>
+												<li data-sourcepos="53:4-53:228">Si la base de datos está en modo solo lectura, lo puedes arreglar ejecutando el siguiente comando para establecer el propietario y grupo en el sistema host: <code>docker exec pialert chown -R www-data:www-data /home/pi/pialert/db/pialert.db</code>.</li>
 												</ul>',
 'HelpFAQ_Cat_General_103_head' => 'La página de inicio de sesión no aparece, incluso después de cambiar la contraseña.',
 'HelpFAQ_Cat_General_103_text' => 'Además de la contraseña, el archivo de configuración debe contener <span class="text-danger help_faq_code">~/pialert/config/pialert.conf</span> 
@@ -412,196 +479,215 @@ $lang['es_es'] = array(
 											 puertos (agrupación de puertos), así como múltiples dispositivos a un puerto (máquinas virtuales).',
 
 //////////////////////////////////////////////////////////////////
-// Settings (based on work of https://github.com/mariorodriguezlopez/Pi.Alert/)
+// Front end events
 //////////////////////////////////////////////////////////////////
 
-'API_settings_group' => '<i class="fa fa-arrow-down-up-across-line"></i> API',	
+'test_event_tooltip' => 'Guarda tus cambios antes de probar nuevos ajustes.',
+'test_event_icon' => 'fa-vial-circle-check',
+'run_event_tooltip' => 'Activa el ajuste y guarda tus cambios antes de ejecutarlo.',
+'run_event_icon' => 'fa-play',
+'general_event_title' => 'Ejecutar un evento ad-hoc',
+'general_event_description' => 'El evento que has ejecutado puede tardar un rato mientras finalizan procesos en segundo plano. La ejecución ha terminado cuando ves <code>finalizado</code> abajo. Comprueba el <a onclick=\'setCache(\"activeMaintenanceTab\", \"tab_Logging_id\")\' href=\"/maintenance.php#tab_Logging\">registro de error</a> si no has obtenido el resultado esperado. <br/> <br/>  Estado: ',
+									 
+//////////////////////////////////////////////////////////////////
+// Plugins
+//////////////////////////////////////////////////////////////////
+
+'Plugins_Unprocessed_Events' => 'Eventos sin procesar',
+'Plugins_Objects' => 'Objetos del Plugin',
+'Plugins_History' => 'Historial de eventos',									 
+					 
+//////////////////////////////////////////////////////////////////
+// Settings
+//////////////////////////////////////////////////////////////////
+
+'settings_missing' => 'Actualiza la página, no todos los ajustes se han cargado. Probablemente sea por una sobrecarga de la base de datos.',
+'settings_missing_block' => 'No puedes guardar los ajustes sin establecer todas las claves. Actualiza la página. Problabmente esté causado por una sobrecarga de la base de datos.',
+'settings_old' => 'Los ajustes mostrados en esta página están desactualizados. Probablemente sea por un escaneo en proceso. Los ajustes se guardan en el archivo <code>pialert.conf</code>, pero el proceso en segundo plano no las ha importado todavía a la base de datos. Puedes esperar a que los ajustes se actualicen para evitar sobreescribirlos con los ajustes antiguos. Si te da igual perder los ajustes desde la última vez que guardaste y ahora, siéntete libre de guardarlos de nuevo. También hay copias de seguridad creadas si necesitas comparar tus ajustes más tarde.',
+'settings_imported' => 'Última vez que los ajustes fueron importados desde el archivo pialert.conf:',
+'settings_expand_all' => 'Expandir todo',											 
 
 // General
-
-'DAYS_TO_KEEP_EVENTS_description' => 'Esta es una configuración de mantenimiento. Esto especifica el número de días de entradas de eventos que se guardarán. Todos los eventos anteriores se eliminarán periódicamente.',
-'DAYS_TO_KEEP_EVENTS_name' => 'Eliminar eventos anteriores a',
-'PIALERT_WEB_PASSWORD_description' => 'La contraseña predeterminada es <code>123456</code>. Para cambiar la contraseña, ejecute <code>/home/pi/pialert/back/pialert-cli</code> en el contenedor',
-'PIALERT_WEB_PASSWORD_name' => 'Contraseña de inicio de sesión',
-'PIALERT_WEB_PROTECTION_description' => 'Cuando está habilitado, se muestra un cuadro de diálogo de inicio de sesión. Lea detenidamente a continuación si se le bloquea el acceso a su instancia.',
-'PIALERT_WEB_PROTECTION_name' => 'Habilitar inicio de sesión',
-'REPORT_DASHBOARD_URL_description' => 'Esta URL se utiliza como base para generar enlaces en los correos electrónicos. Ingrese la URL completa que comienza con <code>http://</code>, incluido el número de puerto (sin barra inclinada al final <code>/</code>).',
-'REPORT_DASHBOARD_URL_name' => 'Pi.Alert URL',
-'REPORT_FROM_description' => 'Asunto del correo electrónico de notificación.',
-'REPORT_FROM_name' => 'Asunto del email',
-'REPORT_MAIL_description' => 'Si está habilitado, se envía un correo electrónico con una lista de cambios a los que se ha suscrito. Complete también todas las configuraciones restantes relacionadas con la configuración de SMTP a continuación',
-'REPORT_MAIL_name' => 'Habilitar email',
-'REPORT_TO_description' => 'Dirección de correo electrónico a la que se enviará la notificación.',
-	'REPORT_TO_name' => 'Enviar el email a',
-	'SCAN_CYCLE_MINUTES_description' => 'El retraso entre escaneos. Si usa arp-scan, el tiempo de escaneo en sí depende de la cantidad de direcciones IP para verificar. Esto está influenciado por la máscara de red configurada en la configuración <a href="#SCAN_SUBNETS"><code>SCAN_SUBNETS</code></a> en la parte superior. Cada IP toma un par de segundos para escanear.',
-	'SCAN_CYCLE_MINUTES_name' => 'Retraso del ciclo de escaneo',
-	'SCAN_SUBNETS_description' => 'El tiempo de escaneo arp en sí depende de la cantidad de direcciones IP para verificar.
-El número de direcciones IP para comprobar depende de la <a target="_blank" href="https://www.calculator.net/ip-subnet-calculator.html">máscara de red</a> que establezca aquí.
-Por ejemplo, una máscara <code>/24</code> da como resultado 256 IP para verificar, mientras que <code>/16</code>
-controles de máscara alrededor de 65,536. Cada IP toma un par de segundos. Esto significa que con una configuración incorrecta
-el arp-scan tardará horas en completarse en lugar de segundos.
-<ol>
-<li>Especifique la máscara de red. Por ejemplo, el filtro <code>192.168.1.0/24</code> cubre los rangos de IP 192.168.1.0 a 192.168.1.255.</li>
-<li>Ejecute <code>ifconfig</code> en su contenedor para encontrar los nombres de su interfaz (por ejemplo: <code>eth0</code>, <code>eth1</code>)</li>
-</ol>
-	',
+'General_display_name' => 'General',
+'General_icon' => '<i class="fa fa-gears"></i>',
+'ENABLE_ARPSCAN_name' => 'Activar escaneo ARP',
+'ENABLE_ARPSCAN_description' => 'El escaneo Arp es una herramienta de la línea de comandos que usa el protocolo ARP para encontrar e identificar la ip de los dispositivos. Una alternativa a este escaneo sería activar los ajustes de la <a onclick="toggleAllSettings()" href="#PIHOLE_ACTIVE"><code>PIHOLE_ACTIVE</code>integración con PiHole</a>.',
 'SCAN_SUBNETS_name' => 'Subredes para escanear',
-'TIMEZONE_description' => 'Zona horaria para mostrar las estadísticas correctamente. Encuentra tu zona horaria<a target="_blank" href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones" rel="nofollow">aquí</a>.',
+'SCAN_SUBNETS_description' => 'El tiempo del escaneo ARP depende del número de ips a comprobar, así que es importante establecer correctamente la máscara y la interfaz de red. Comprueba la <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SUBNETS.md" target="_blank">documentación sobre sudredes</a> para obtener ayuda para establecer VLANs, cuáles son soportadas o como averiguar la máscara y la interfaz de red.',
+'LOG_LEVEL_name' => 'Imprimir registros adicionales',
+'LOG_LEVEL_description' => 'Esto hará que el registro tenga más información. Util para depurar que eventos se van guardando en la base de datos.',
 'TIMEZONE_name' => 'Zona horaria',
-'UI_LANG_description' => 'Seleccione el idioma de interfaz de usuario preferido.',
+'TIMEZONE_description' => 'La zona horaria para mostrar las estadísticas correctamente. Encuentra tu zona horaria <a target="_blank" href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones" rel="nofollow">aquí</a>.',
+'ENABLE_PLUGINS_name' => 'Habilitar complementos',
+'ENABLE_PLUGINS_description' => 'Habilita la funcionalidad de los <a target="_blank" href="https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins">complementos</a>. Cargar los complementos requiere más recursos de hardware, así que quizás quieras desactivarlo en hardware poco potente.',
+'PIALERT_WEB_PROTECTION_name' => 'Habilitar inicio de sesión',
+'PIALERT_WEB_PROTECTION_description' => 'Cuando está habilitado, se muestra un cuadro de diálogo de inicio de sesión. Lea detenidamente a continuación si se le bloquea el acceso a su instancia.',
+'PIALERT_WEB_PASSWORD_name' => 'Contraseña de inicio de sesión',
+'PIALERT_WEB_PASSWORD_description' => 'La contraseña predeterminada es <code>123456</code>. Para cambiar la contraseña, ejecute <code>/home/pi/pialert/back/pialert-cli</code> en el contenedor',
+'INCLUDED_SECTIONS_name' => 'Notificar en',
+'INCLUDED_SECTIONS_description' => 'Especifica que eventos envían notificaciones. Elimina los tipos de eventos de los que no quieras recibir notificaciones. Este ajuste sobreescribe los ajustes específicos de los dispositivos en la interfaz. (<code>CTRL + Clic</code> para seleccionar / deseleccionar).',
+'SCAN_CYCLE_MINUTES_name' => 'Retraso del ciclo de escaneo',
+'SCAN_CYCLE_MINUTES_description' => 'El retraso entre escaneos. Si usa arp-scan, el tiempo de escaneo en sí depende de la cantidad de direcciones IP para verificar. Esto está influenciado por la máscara de red configurada en la configuración <a href="#SCAN_SUBNETS"><code>SCAN_SUBNETS</code></a> en la parte superior. Cada IP toma un par de segundos para escanear.',
+'DAYS_TO_KEEP_EVENTS_name' => 'Eliminar eventos anteriores a',
+'DAYS_TO_KEEP_EVENTS_description' => 'Esta es una configuración de mantenimiento. Esto especifica el número de días de entradas de eventos que se guardarán. Todos los eventos anteriores se eliminarán periódicamente.',
+'REPORT_DASHBOARD_URL_name' => 'URL de Pi.Alert',
+'REPORT_DASHBOARD_URL_description' => 'Esta URL se utiliza como base para generar enlaces en los correos electrónicos. Ingrese la URL completa que comienza con <code>http://</code>, incluido el número de puerto (sin barra inclinada al final <code>/</code>).',
+'DIG_GET_IP_ARG_name' => 'Descubrir de IP de Internet',
+'DIG_GET_IP_ARG_description' => 'Cambie los argumentos de la <a href="https://linux.die.net/man/1/dig" target="_blank">utilidad de dig</a> si tiene problemas para resolver su IP de Internet. Los argumentos se agregan al final del siguiente comando: <code>dig +short </code>.',
 'UI_LANG_name' => 'Idioma de interfaz',
+'UI_LANG_description' => 'Seleccione el idioma de interfaz de usuario preferido.',
+'UI_PRESENCE_name' => 'Mostrar en el gráfico de presencia',
+'UI_PRESENCE_description' => 'Elige que estados del dispositivo deben mostrarse en la gráfica de <b>Presencia del dispositivo a lo largo del tiempo</b>  de la página de <a href="/devices.php" target="_blank">Dispositivos</a>. (<code>CTRL + Clic</code> para seleccionar / deseleccionar)',
 
 // Email
-
-'SMTP_FORCE_SSL_description' => 'Forzar SSL al conectarse a su servidor SMTP',
-'SMTP_FORCE_SSL_name' => 'Forzar SSL',
-'SMTP_PASS_description' => 'La contraseña del servidor SMTP.',
-'SMTP_PASS_name' => 'SMTP password',
-'SMTP_PORT_description' => 'Número de puerto utilizado para la conexión SMTP. Establézcalo en <code>0</code> si no desea utilizar un puerto al conectarse al servidor SMTP.',
-'SMTP_PORT_name' => 'SMTP server PORT',
+'Email_display_name' => 'Email',
+'Email_icon' => '<i class="fa fa-at"></i>',
+'REPORT_MAIL_name' => 'Habilitar email',
+'REPORT_MAIL_description' => 'Si está habilitado, se envía un correo electrónico con una lista de cambios a los que se ha suscrito. Complete también todas las configuraciones restantes relacionadas con la configuración de SMTP a continuación',
+'SMTP_SERVER_name' => 'URL del servidor SMTP',
 'SMTP_SERVER_description' => 'La URL del host del servidor SMTP. Por ejemplo, <code>smtp-relay.sendinblue.com</code>. Para utilizar Gmail como servidor SMTP <a target="_blank" href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/SMTP_GMAIL.md">siga esta guía</a >',
-'SMTP_SERVER_name' => 'SMTP server URL',
-'SMTP_SKIP_LOGIN_description' => 'No utilice la autenticación cuando se conecte al servidor SMTP.',
+'SMTP_PORT_name' => 'Puerto del servidor SMTP',
+'SMTP_PORT_description' => 'Número de puerto utilizado para la conexión SMTP. Establézcalo en <code>0</code> si no desea utilizar un puerto al conectarse al servidor SMTP.',
 'SMTP_SKIP_LOGIN_name' => 'Omitir autenticación',
-'SMTP_SKIP_TLS_description' => 'Deshabilite TLS cuando se conecte a su servidor SMTP.',
-'SMTP_SKIP_TLS_name' => 'No usar TLS',
+'SMTP_SKIP_LOGIN_description' => 'No utilice la autenticación cuando se conecte al servidor SMTP.',
+'SMTP_USER_name' => 'Nombre de usuario SMTP',
 'SMTP_USER_description' => 'El nombre de usuario utilizado para iniciar sesión en el servidor SMTP (a veces, una dirección de correo electrónico completa).',
-'SMTP_USER_name' => 'SMTP user',
-
-//API
-
-'API_CUSTOM_SQL_description' => 'Puede especificar una consulta SQL personalizada que generará un archivo JSON y luego lo expondrá a través del <a href="/api/table_custom_endpoint.json" target="_blank">archivo <code>table_custom_endpoint.json</code ></a>.',
-'API_CUSTOM_SQL_name' => 'Endpoint personalizado',
-
-
+'SMTP_PASS_name' => 'Contraseña de SMTP',
+'SMTP_PASS_description' => 'La contraseña del servidor SMTP.',
+'SMTP_SKIP_TLS_name' => 'No usar TLS',
+'SMTP_SKIP_TLS_description' => 'Deshabilite TLS cuando se conecte a su servidor SMTP.',
+'SMTP_FORCE_SSL_name' => 'Forzar SSL',
+'SMTP_FORCE_SSL_description' => 'Forzar SSL al conectarse a su servidor SMTP',
+'REPORT_TO_name' => 'Enviar el email a',
+'REPORT_TO_description' => 'Dirección de correo electrónico a la que se enviará la notificación.',
+'REPORT_FROM_name' => 'Asunto del email',
+'REPORT_FROM_description' => 'Asunto del correo electrónico de notificación.',
+									  
+// Webhooks
+'Webhooks_display_name' => 'Webhooks',
+'Webhooks_icon' => '<i class="fa fa-circle-nodes"></i>',
+'REPORT_WEBHOOK_name' => 'Habilitar webhooks',
+'REPORT_WEBHOOK_description' => 'Habilite webhooks para notificaciones. Los webhooks lo ayudan a conectarse a muchas herramientas de terceros, como IFTTT, Zapier o <a href="https://n8n.io/" target="_blank">n8n</a>, por nombrar algunas. Consulte esta sencilla <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/WEBHOOK_N8N.md" target="_blank">guía de n8n aquí</a> para obtener comenzó. Si está habilitado, configure los ajustes relacionados a continuación.',
+'WEBHOOK_URL_name' => 'URL de destino',
+'WEBHOOK_URL_description' => 'URL de destino comienza con <code>http://</code> o <code>https://</code>.',
+'WEBHOOK_PAYLOAD_name' => 'Tipo de carga',
+'WEBHOOK_PAYLOAD_description' => 'El formato de datos de carga de Webhook para el atributo <code>body</code> > <code>attachments</code> > <code>text</code> en el json de carga. Vea un ejemplo de la carga <a target="_blank" href="https://github.com/jokob-sk/Pi.Alert/blob/main/back/webhook_json_sample.json">aquí</a>. (por ejemplo: para discord use <code>html</code>)',
+'WEBHOOK_REQUEST_METHOD_name' => 'Método de solicitud',
+'WEBHOOK_REQUEST_METHOD_description' => 'El método de solicitud HTTP que se utilizará para la llamada de webhook.',
+'Webhooks_settings_group' => '<i class="fa fa-circle-nodes"></i> Webhooks',	
+																				 																				  
 // Apprise
-
+'Apprise_display_name' => 'Apprise',
+'Apprise_icon' => '<i class="fa fa-bullhorn"></i>',
+'REPORT_APPRISE_name' => 'Habilitar Apprise',
+'REPORT_APPRISE_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://hub.docker.com/r/caronc/apprise">Apprise</a>.',
 'APPRISE_HOST_description' => 'Apprise host URL que comienza con <code>http://</code> o <code>https://</code>. (no olvide incluir <code>/notify</code> al final)',
 'APPRISE_HOST_name' => 'Apprise host URL',
-'APPRISE_PAYLOAD_description' => 'Seleccione el tipo de carga útil enviada a Apprise. Por ejemplo, <code>html</code> funciona bien con correos electrónicos, <code>text</code> con aplicaciones de chat, como Telegram.',
-'APPRISE_PAYLOAD_name' => 'Tipo de carga',
-'APPRISE_URL_description' => 'Informar de la URL de destino de la notificación. Por ejemplo, para Telegram sería <code>tgram://{bot_token}/{chat_id}</code>.',
 'APPRISE_URL_name' => 'URL de notificación de Apprise',
-
-// Pushsafer
-'REPORT_PUSHSAFER_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://www.pushsafer.com/">Pushsafer</a>.',
-'REPORT_PUSHSAFER_name' => 'Habilitar Pushsafer',
-
-
-//DYNDNS
-
-
-'DDNS_ACTIVE_name' => 'Habilitar DynDNS',
-'DDNS_DOMAIN_name' => 'URL del dominio DynDNS',
-'DDNS_PASSWORD_name' => 'DynDNS password',
-'DDNS_UPDATE_URL_description' => 'Actualice la URL que comienza con <code>http://</code> o <code>https://</code>.',
-'DDNS_UPDATE_URL_name' => 'DynDNS update URL',
-'DDNS_USER_name' => 'DynDNS user',
-'DHCP_ACTIVE_description' => 'Debe asignar <code>:/etc/pihole/dhcp.leases</code> en el archivo <code>docker-compose.yml</code> si habilita esta configuración.',
-'DHCP_ACTIVE_name' => 'Habilitar PiHole DHCP',
-'DIG_GET_IP_ARG_description' => 'Cambie los argumentos de la <a href="https://linux.die.net/man/1/dig" target="_blank">utilidad de dig</a> si tiene problemas para resolver su IP de Internet. Los argumentos se agregan al final del siguiente comando: <code>dig +short </code>.',
-'DIG_GET_IP_ARG_name' => 'Descubrir de IP de Internet',
-
-// MQTT
-'REPORT_MQTT_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://www.home-assistant.io/integrations/mqtt/">MQTT</a> a su Home Assistance.',
-'REPORT_MQTT_name' => 'Habilitar MQTT',
-'MQTT_BROKER_description' => 'URL del host MQTT (no incluya <code>http://</code> o <code>https://</code>).',
-'MQTT_BROKER_name' => 'MQTT broker URL',
-'MQTT_DELAY_SEC_description' => 'Un pequeño truco: retrase la adición a la cola en caso de que el proceso se reinicie y los procesos de publicación anteriores se anulen (se necesitan ~<code>2</code>s para actualizar la configuración de un sensor en el intermediario). Probado con <code>2</code>-<code>3</code> segundos de retraso. Este retraso solo se aplica cuando se crean dispositivos (durante el primer bucle de notificación). No afecta los escaneos o notificaciones posteriores.',
-'MQTT_DELAY_SEC_name' => 'Retraso de MQTT por dispositivo',
-'MQTT_PASSWORD_description' => 'Contraseña utilizada para iniciar sesión en su instancia de agente de MQTT.',
-'MQTT_PASSWORD_name' => 'MQTT password',
-'MQTT_PORT_description' => 'Puerto donde escucha el broker MQTT. Normalmente <code>1883</code>.',
-'MQTT_PORT_name' => 'MQTT broker puerto',
-'MQTT_QOS_description' => 'Configuración de calidad de servicio para el envío de mensajes MQTT. <code>0</code>: baja calidad a <code>2</code>: alta calidad. Cuanto mayor sea la calidad, mayor será el retraso.',
-'MQTT_QOS_name' => 'Calidad de servicio MQTT',
-'MQTT_USER_description' => 'Nombre de usuario utilizado para iniciar sesión en su instancia de agente de MQTT.',
-'MQTT_USER_name' => 'MQTT user',
-'MQTT_settings_group' => '<i class="fa fa-square-rss"></i> MQTT',
-
-// NMAP 
-
-'NMAP_ACTIVE_description' => 'Si está habilitado, ejecutará un escaneo en un dispositivo recién encontrado. Para un análisis programado o único, verifique la configuración de <a href="#NMAP_RUN"><code>NMAP_RUN</code></a>.',
-'NMAP_ACTIVE_name' => 'Ejecución del ciclo',
-'NMAP_ARGS_description' => 'Argumentos utilizados para ejecutar el análisis de Nmap. Tenga cuidado de especificar <a href="https://linux.die.net/man/1/nmap" target="_blank">los argumentos</a> correctamente. Por ejemplo, <code>-p -10000</code> escanea los puertos del 1 al 10000.',
-'NMAP_ARGS_name' => 'Argumentos',
-'NMAP_RUN_SCHD_description' => 'Solo está habilitado si selecciona <code>programar</code> en la configuración de <a href="#NMAP_RUN"><code>NMAP_RUN</code></a>. Asegúrese de ingresar el cronograma en el formato tipo cron correcto.',
-'NMAP_RUN_SCHD_name' => 'Programar',
-'NMAP_RUN_description' => 'Habilite un escaneo regular de Nmap en su red en todos los dispositivos. Los ajustes de programación se pueden encontrar a continuación. Si selecciona <code>una vez</code>, Nmap se ejecuta solo una vez al inicio durante el tiempo especificado en la configuración de <a href="#NMAP_TIMEOUT"><code>NMAP_TIMEOUT</code></a>.',
-'NMAP_RUN_name' => 'Ejecución programada',
-'NMAP_TIMEOUT_description' => 'Tiempo máximo en segundos para esperar a que finalice un escaneo de Nmap en cualquier dispositivo.',
+'APPRISE_URL_description' => 'Informar de la URL de destino de la notificación. Por ejemplo, para Telegram sería <code>tgram://{bot_token}/{chat_id}</code>.',
 
 // NTFY
-'REPORT_NTFY_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://ntfy.sh/">NTFY</a>.',
+'NTFY_display_name' => 'NTFY',
+'NTFY_icon' => '<i class="fa fa-terminal"></i>',
 'REPORT_NTFY_name' => 'Habilitar NTFY',
-'NTFY_HOST_description' => 'URL de host NTFY que comienza con <code>http://</code> o <code>https://</code>. Puede usar la instancia alojada en <a target="_blank" href="https://ntfy.sh/">https://ntfy.sh</a> simplemente ingresando <code>https://ntfy. sh</código>.',
+'REPORT_NTFY_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://ntfy.sh/">NTFY</a>.',
 'NTFY_HOST_name' => 'NTFY host URL',
-'NTFY_PASSWORD_description' => 'Ingrese la contraseña si necesita (host) una instancia con autenticación habilitada.',
-'NTFY_PASSWORD_name' => 'NTFY password',
-'NTFY_TOPIC_name' => 'NTFY topic',
+'NTFY_HOST_description' => 'URL de host NTFY que comienza con <code>http://</code> o <code>https://</code>. Puede usar la instancia alojada en <a target="_blank" href="https://ntfy.sh/">https://ntfy.sh</a> simplemente ingresando <code>https://ntfy. sh</código>.',
+'NTFY_TOPIC_name' => 'Tema de NTFY',
+'NTFY_TOPIC_description' => 'Tu tema secreto.',
+'NTFY_USER_name' => 'Usuario de NTFY',
 'NTFY_USER_description' => 'Ingrese usuario si necesita (alojar) una instancia con autenticación habilitada.',
-'NTFY_USER_name' => 'NTFY user',
-'NTFY_settings_group' => '<i class="fa fa-terminal"></i> NTFY',
+'NTFY_PASSWORD_name' => 'Contraseña de NTFY',
+'NTFY_PASSWORD_description' => 'Ingrese la contraseña si necesita (host) una instancia con autenticación habilitada.',
 
-// Pholus
+// Pushsafer
+'PUSHSAFER_display_name' => 'Pushsafer',
+'PUSHSAFER_icon' => '<i class="fa fa-bell"></i>',
+'REPORT_PUSHSAFER_name' => 'Habilitar Pushsafer',
+'REPORT_PUSHSAFER_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://www.pushsafer.com/">Pushsafer</a>.',
+'PUSHSAFER_TOKEN_name' => 'Token de Pushsafer',
+'PUSHSAFER_TOKEN_description' => 'Su clave secreta de la API de Pushsafer (token).',
+'APPRISE_PAYLOAD_name' => 'Tipo de carga',
+'APPRISE_PAYLOAD_description' => 'Seleccione el tipo de carga útil enviada a Apprise. Por ejemplo, <code>html</code> funciona bien con correos electrónicos, <code>text</code> con aplicaciones de chat, como Telegram.',
 
-'Pholus_settings_group' => '<i class="fa fa-search"></i> Pholus',
-'PHOLUS_ACTIVE_description' => '<a href="https://github.com/jokob-sk/Pi.Alert/tree/main/pholus" target="_blank" >Pholus</a> es una herramienta de rastreo para descubrir información adicional sobre los dispositivos en la red, incluido el nombre del dispositivo. Si está habilitado, ejecutará el escaneo antes de cada ciclo de escaneo de red hasta que no haya dispositivos <code>(unknown)</code> o <code>(name not found)</code>. Tenga en cuenta que puede enviar spam a la red con tráfico innecesario. Depende de la configuración de <a onclick="toggleAllSettings()" href="#SCAN_SUBNETS"><code>SCAN_SUBNETS</code></a>. Para un análisis programado o único, verifique la configuración de <a href="#PHOLUS_RUN"><code>PHOLUS_RUN</code></a>.',
-'PHOLUS_ACTIVE_name' => 'Ejecución del ciclo',
-'PHOLUS_DAYS_DATA_description' => 'Cuántos días de entradas de escaneo de Pholus deben conservarse (globalmente, ¡no específico del dispositivo!). El archivo <a href="/maintenance.php#tab_Logging">pialert_pholus.log</a> no se modifica. Introduzca <code>0</code> para desactivar.',
-'PHOLUS_DAYS_DATA_name' => 'Retención de datos',
-'PHOLUS_FORCE_description' => 'Fuerce el escaneo de cada escaneo de red, incluso si no hay dispositivos <code>(unknown)</code> o <code>(name not found)</code>. Tenga cuidado al habilitar esto, ya que la detección puede inundar fácilmente su red.',
-'PHOLUS_FORCE_name' => 'Escaneo de fuerza de ciclo',
-'PHOLUS_RUN_SCHD_description' => 'Solo está habilitado si selecciona <code>programar</code> en la configuración de <a href="#PHOLUS_RUN"><code>PHOLUS_RUN</code></a>. Asegúrese de ingresar el horario en el formato similar a cron correcto
-(por ejemplo, validar en <a href="https://crontab.guru/" target="_blank">crontab.guru</a>). Por ejemplo, ingresar <code>0 4 * * *</code> ejecutará el escaneo después de las 4 am en el <a onclick="toggleAllSettings()" href="#TIMEZONE"><code>TIMEZONE</code> que configuró arriba</a>. Se ejecutará la PRÓXIMA vez que pase el tiempo.',
-'PHOLUS_RUN_SCHD_name' => 'Programar',
-'PHOLUS_RUN_TIMEOUT_description' => 'El tiempo de espera en segundos para el escaneo Pholus programado. Se aplican las mismas notas con respecto a la duración que en la configuración de <a href="#PHOLUS_TIMEOUT"><code>PHOLUS_TIMEOUT</code></a>. Un escaneo programado no verifica si hay dispositivos <code>(unknown)</code> o <code>(name not found)</code>, el escaneo se ejecuta de cualquier manera.',
-'PHOLUS_RUN_TIMEOUT_name' => 'Tiempo de espera de ejecución programado',
-'PHOLUS_RUN_description' => 'Habilite un escaneo regular de Pholus en su red. Los ajustes de programación se pueden encontrar a continuación. Si selecciona <code>una vez</code>, Pholus se ejecuta solo una vez al inicio durante el tiempo especificado en la configuración de <a href="#PHOLUS_RUN_TIMEOUT"><code>PHOLUS_RUN_TIMEOUT</code></a>.',
-'PHOLUS_RUN_name' => 'Ejecución programada',
-'PHOLUS_TIMEOUT_description' => '¿Cuánto tiempo en segundos debe rastrear Pholus en cada interfaz si se cumple la condición anterior? Cuanto más tiempo lo deje encendido, es más probable que los dispositivos transmitan más información. Este tiempo de espera se suma al tiempo que lleva realizar un escaneo arp en su red.',
-'PHOLUS_TIMEOUT_name' => 'Tiempo de espera de ciclo',
+// MQTT
+'MQTT_display_name' => 'MQTT',
+'MQTT_icon' => '<i class="fa fa-square-rss"></i>',
+'REPORT_MQTT_name' => 'Habilitar MQTT',
+'REPORT_MQTT_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://www.home-assistant.io/integrations/mqtt/">MQTT</a> a su Home Assistance.',
+'MQTT_BROKER_name' => 'URL del broker MQTT',
+'MQTT_BROKER_description' => 'URL del host MQTT (no incluya <code>http://</code> o <code>https://</code>).',
+'MQTT_PORT_name' => 'Puerto del broker MQTT',
+'MQTT_PORT_description' => 'Puerto donde escucha el broker MQTT. Normalmente <code>1883</code>.',
+'MQTT_USER_name' => 'Usuario de MQTT',
+'MQTT_USER_description' => 'Nombre de usuario utilizado para iniciar sesión en su instancia de agente de MQTT.',
+'MQTT_PASSWORD_name' => 'Contraseña de MQTT',
+'MQTT_PASSWORD_description' => 'Contraseña utilizada para iniciar sesión en su instancia de agente de MQTT.',
+'MQTT_QOS_name' => 'Calidad de servicio MQTT',
+'MQTT_QOS_description' => 'Configuración de calidad de servicio para el envío de mensajes MQTT. <code>0</code>: baja calidad a <code>2</code>: alta calidad. Cuanto mayor sea la calidad, mayor será el retraso.',
+'MQTT_DELAY_SEC_name' => 'Retraso de MQTT por dispositivo',
+'MQTT_DELAY_SEC_description' => 'Un pequeño truco: retrase la adición a la cola en caso de que el proceso se reinicie y los procesos de publicación anteriores se anulen (se necesitan ~<code>2</code>s para actualizar la configuración de un sensor en el intermediario). Probado con <code>2</code>-<code>3</code> segundos de retraso. Este retraso solo se aplica cuando se crean dispositivos (durante el primer bucle de notificación). No afecta los escaneos o notificaciones posteriores.',
+
+//DYNDNS
+'DynDNS_display_name' => 'DynDNS',
+'DynDNS_icon' => '<i class="fa fa-globe"></i>',
+'DDNS_ACTIVE_name' => 'Habilitar DynDNS',
+'DDNS_ACTIVE_description' => '',
+'DDNS_DOMAIN_name' => 'URL del dominio DynDNS',
+'DDNS_DOMAIN_description' => '',
+'DDNS_USER_name' => 'Usuario de DynDNS',
+'DDNS_USER_description' => '',
+'DDNS_PASSWORD_name' => 'Contraseña de DynDNS',
+'DDNS_PASSWORD_description' => '',
+'DDNS_UPDATE_URL_name' => 'URL de actualización de DynDNS',
+'DDNS_UPDATE_URL_description' => 'Actualice la URL que comienza con <code>http://</code> o <code>https://</code>.',
 
 // PiHole
-
-'PiHole_settings_group' => '<i class="fa fa-seedling"></i> PiHole',
-'PIHOLE_ACTIVE_description' => 'Debe mapear <code>:/etc/pihole/pihole-FTL.db</code> en el archivo <code>docker-compose.yml</code> si habilita esta configuración.',
+'PiHole_display_name' => 'PiHole',
+'PiHole_icon' => '<i class="fa fa-seedling"></i>',
 'PIHOLE_ACTIVE_name' => 'Habilitar el mapeo de PiHole',
-'PRINT_LOG_description' => 'Esta configuración habilitará un registro más detallado. Útil para depurar eventos que se escriben en la base de datos.',
-'PRINT_LOG_name' => 'Imprimir registro adicional',
-'PUSHSAFER_TOKEN_description' => 'Su clave secreta de la API de Pushsafer (token).',
-'PUSHSAFER_TOKEN_name' => 'Pushsafer token',
-'PUSHSAFER_settings_group' => '<i class="fa fa-bell"></i> Pushsafer',
+'PIHOLE_ACTIVE_description' => 'Debe mapear <code>:/etc/pihole/pihole-FTL.db</code> en el archivo <code>docker-compose.yml</code> si habilita esta configuración.',
+'DHCP_ACTIVE_name' => 'Habilitar PiHole DHCP',
+'DHCP_ACTIVE_description' => 'Debe asignar <code>:/etc/pihole/dhcp.leases</code> en el archivo <code>docker-compose.yml</code> si habilita esta configuración.',
 
-//Apprise
+// Pholus
+'Pholus_display_name' => 'Pholus',
+'Pholus_icon' => '<i class="fa fa-search"></i>',
+'PHOLUS_ACTIVE_name' => 'Ejecución del ciclo',
+'PHOLUS_ACTIVE_description' => '<a href="https://github.com/jokob-sk/Pi.Alert/tree/main/pholus" target="_blank" >Pholus</a> es una herramienta de rastreo para descubrir información adicional sobre los dispositivos en la red, incluido el nombre del dispositivo. Si está habilitado, ejecutará el escaneo antes de cada ciclo de escaneo de red hasta que no haya dispositivos <code>(unknown)</code> o <code>(name not found)</code>. Tenga en cuenta que puede enviar spam a la red con tráfico innecesario. Depende de la configuración de <a onclick="toggleAllSettings()" href="#SCAN_SUBNETS"><code>SCAN_SUBNETS</code></a>. Para un análisis programado o único, verifique la configuración de <a href="#PHOLUS_RUN"><code>PHOLUS_RUN</code></a>.',
+'PHOLUS_TIMEOUT_name' => 'Tiempo de espera de ciclo',
+'PHOLUS_TIMEOUT_description' => '¿Cuánto tiempo en segundos debe rastrear Pholus en cada interfaz si se cumple la condición anterior? Cuanto más tiempo lo deje encendido, es más probable que los dispositivos transmitan más información. Este tiempo de espera se suma al tiempo que lleva realizar un escaneo arp en su red.',
+'PHOLUS_FORCE_name' => 'Escaneo de fuerza de ciclo',
+'PHOLUS_FORCE_description' => 'Fuerce el escaneo de cada escaneo de red, incluso si no hay dispositivos <code>(unknown)</code> o <code>(name not found)</code>. Tenga cuidado al habilitar esto, ya que la detección puede inundar fácilmente su red.',
+'PHOLUS_RUN_name' => 'Ejecución programada',
+'PHOLUS_RUN_description' => 'Habilite un escaneo regular de Pholus en su red. Los ajustes de programación se pueden encontrar a continuación. Si selecciona <code>una vez</code>, Pholus se ejecuta solo una vez al inicio durante el tiempo especificado en la configuración de <a href="#PHOLUS_RUN_TIMEOUT"><code>PHOLUS_RUN_TIMEOUT</code></a>.',
+'PHOLUS_RUN_TIMEOUT_name' => 'Tiempo de espera de ejecución programado',
+'PHOLUS_RUN_TIMEOUT_description' => 'El tiempo de espera en segundos para el escaneo Pholus programado. Se aplican las mismas notas con respecto a la duración que en la configuración de <a href="#PHOLUS_TIMEOUT"><code>PHOLUS_TIMEOUT</code></a>. Un escaneo programado no verifica si hay dispositivos <code>(unknown)</code> o <code>(name not found)</code>, el escaneo se ejecuta de cualquier manera.',
+'PHOLUS_RUN_SCHD_name' => 'Programar',
+'PHOLUS_RUN_SCHD_description' => 'Solo está habilitado si selecciona <code>programar</code> en la configuración de <a href="#PHOLUS_RUN"><code>PHOLUS_RUN</code></a>. Asegúrese de ingresar el horario en el formato similar a cron correcto
+(por ejemplo, validar en <a href="https://crontab.guru/" target="_blank">crontab.guru</a>). Por ejemplo, ingresar <code>0 4 * * *</code> ejecutará el escaneo después de las 4 am en el <a onclick="toggleAllSettings()" href="#TIMEZONE"><code>TIMEZONE</code> que configuró arriba</a>. Se ejecutará la PRÓXIMA vez que pase el tiempo.',
+'PHOLUS_DAYS_DATA_name' => 'Retención de datos',
+'PHOLUS_DAYS_DATA_description' => 'Cuántos días de entradas de escaneo de Pholus deben conservarse (globalmente, ¡no específico del dispositivo!). El archivo <a href="/maintenance.php#tab_Logging">pialert_pholus.log</a> no se modifica. Introduzca <code>0</code> para desactivar.',
 
-'REPORT_APPRISE_description' => 'Habilitar el envío de notificaciones a través de <a target="_blank" href="https://hub.docker.com/r/caronc/apprise">Apprise</a>.',
-'REPORT_APPRISE_name' => 'Habilitar Apprise',
+// NMAP 
+'Nmap_display_name' => 'Nmap',
+'Nmap_icon' => '<i class="fa fa-ethernet"></i>',
+'NMAP_ACTIVE_name' => 'Ejecución del ciclo',
+'NMAP_ACTIVE_description' => 'Si está habilitado, ejecutará un escaneo en un dispositivo recién encontrado. Para un análisis programado o único, verifique la configuración de <a href="#NMAP_RUN"><code>NMAP_RUN</code></a>.',
+'PHOLUS_TIMEOUT_name' => 'Tiempo de espera de ciclo',
+'NMAP_TIMEOUT_description' => 'Tiempo máximo en segundos para esperar a que finalice un escaneo de Nmap en cualquier dispositivo.',
+'NMAP_RUN_name' => 'Ejecución programada',
+'NMAP_RUN_description' => 'Habilite un escaneo regular de Nmap en su red en todos los dispositivos. Los ajustes de programación se pueden encontrar a continuación. Si selecciona <code>una vez</code>, Nmap se ejecuta solo una vez al inicio durante el tiempo especificado en la configuración de <a href="#NMAP_TIMEOUT"><code>NMAP_TIMEOUT</code></a>.',
+'NMAP_RUN_SCHD_name' => 'Programar',
+'NMAP_RUN_SCHD_description' => 'Solo está habilitado si selecciona <code>programar</code> en la configuración de <a href="#NMAP_RUN"><code>NMAP_RUN</code></a>. Asegúrese de ingresar el cronograma en el formato tipo cron correcto.',
+'NMAP_ARGS_name' => 'Argumentos',
+'NMAP_ARGS_description' => 'Argumentos utilizados para ejecutar el análisis de Nmap. Tenga cuidado de especificar <a href="https://linux.die.net/man/1/nmap" target="_blank">los argumentos</a> correctamente. Por ejemplo, <code>-p -10000</code> escanea los puertos del 1 al 10000.',
 
-// Webhooks
-'REPORT_WEBHOOK_description' => 'Habilite webhooks para notificaciones. Los webhooks lo ayudan a conectarse a muchas herramientas de terceros, como IFTTT, Zapier o <a href="https://n8n.io/" target="_blank">n8n</a>, por nombrar algunas. Consulte esta sencilla <a href="https://github.com/jokob-sk/Pi.Alert/blob/main/docs/WEBHOOK_N8N.md" target="_blank">guía de n8n aquí</a> para obtener comenzó. Si está habilitado, configure los ajustes relacionados a continuación.',
-'REPORT_WEBHOOK_name' => 'Habilitar webhooks',
-'WEBHOOK_PAYLOAD_description' => 'El formato de datos de carga de Webhook para el atributo <code>body</code> > <code>attachments</code> > <code>text</code> en el json de carga. Vea un ejemplo de la carga <a target="_blank" href="https://github.com/jokob-sk/Pi.Alert/blob/main/back/webhook_json_sample.json">aquí</a>. (por ejemplo: para discord use <code>html</code>)',
-'WEBHOOK_PAYLOAD_name' => 'Tipo de carga',
-'WEBHOOK_REQUEST_METHOD_description' => 'El método de solicitud HTTP que se utilizará para la llamada de webhook.',
-'WEBHOOK_REQUEST_METHOD_name' => 'Método de solicitud',
-'WEBHOOK_URL_description' => 'URL de destino comienza con <code>http://</code> o <code>https://</code>.',
-'WEBHOOK_URL_name' => 'URL de destino',
-'Webhooks_settings_group' => '<i class="fa fa-circle-nodes"></i> Webhooks',	
+//API
+'API_display_name' => 'API',
+'API_icon' => '<i class="fa fa-arrow-down-up-across-line"></i>',									  
+'API_CUSTOM_SQL_name' => 'Endpoint personalizado',
+'API_CUSTOM_SQL_description' => 'Puede especificar una consulta SQL personalizada que generará un archivo JSON y luego lo expondrá a través del <a href="/api/table_custom_endpoint.json" target="_blank">archivo <code>table_custom_endpoint.json</code ></a>.',
 	
-// Other
-
-'general_event_description' => 'El evento que ha activado puede tardar un tiempo hasta que finalicen los procesos en segundo plano. La ejecución terminó una vez que vea <code>finished</code> a continuación. Consulte el <a onclick=\'setCache("activeMaintenanceTab", "tab_Logging_id")\' href="/maintenance.php#tab_Logging">registro de errores</a> si no obtuvo el resultado esperado. <br/> <br/> Estado:',
-'general_event_title' => 'Ejecución de un evento ad-hoc',
-'run_event_icon' => 'fa-play',
-'run_event_tooltip' => 'Habilite la configuración y guarde sus cambios al principio antes de ejecutarlo.',
-'settings_expand_all' => 'Expandir todo',
-'settings_imported' => 'La última vez que se importó la configuración desde el archivo pialert.conf:',
-'settings_missing' => 'No se han cargado todos los ajustes, actualice la página. Esto probablemente se deba a una gran carga en la base de datos.',
-'settings_missing_block' => 'No puede guardar su configuración sin especificar todas las claves de configuración. Recarga la página. Esto probablemente se deba a una gran carga en la base de datos.',
-'settings_old' => 'La configuración en la base de datos (que se muestra en esta página) está desactualizada. Esto probablemente se deba a un análisis en ejecución. La configuración se guardó en el archivo <code>pialert.conf</code>, pero el proceso en segundo plano aún no tuvo tiempo de importarlo a la base de datos. Puede esperar hasta que la configuración se actualice para no sobrescribir sus valores anteriores. Siéntase libre de guardar su configuración de cualquier manera si no le importa perder la configuración entre la última vez que guardó y ahora. También se crean archivos de respaldo si necesita comparar su configuración más adelante.',
-'test_event_icon' => 'fa-vial-circle-check',
-'test_event_tooltip' => 'Guarde sus cambios antes de probar su configuración.',
-
 );
-?>
+?>

front/plugins.php

@@ -129,9 +129,7 @@ function localize (obj, key) {
     {
         for(i=0;i<obj[key].length;i++)
         {
-            code = obj[key][i]["language_code"]
-
-            // console.log(code)
+            code = obj[key][i]["language_code"]            
 
             if( code == 'en_us')
             {
@@ -146,7 +144,7 @@ function localize (obj, key) {
         }
     }
 
-    result == "" ? en_us : result ;
+    result == "" ? result = en_us : result ;
 
     return result;
 }
@@ -190,6 +188,9 @@ function generateTabs()
     activetab = 'active'
 
     $.each(pluginDefinitions, function(index, obj) {
+
+        // console.log(obj)
+
         $('#tabs-location').append(
             `<li class=" ${activetab}">
                 <a href="#${obj.unique_prefix}" data-plugin-prefix="${obj.unique_prefix}" id="${obj.unique_prefix}_id" data-toggle="tab" >

front/plugins/README.md

@@ -1,17 +1,9 @@
-# ⚠ Disclaimer
-
-Highly experimental feature. Follow the below very carefully and check example plugin(s). Plugin UI is not my priority right now, happy to approve PRs if you are interested in extending/improvintg the UI experience (e.g. making the tables sortable/filterable). 
-
-## ❗ Known issues:
-
-These issues will be hopefully fixed with time, so please don't report them. Instead, if you know how, feel free to investigate and submit a PR to fix the below. Keep the PRs small as it's easier to approve them:
-
-* Existing plugin objects sometimes not interpreted correctly and a new object is created instead, resulting in duplicate entries.
-* Occasional (experienced twice) hanging of processing plugin script file.
-* UI displaying outdated values until the API endpoints get refreshed. 
-
 ## Overview
 
+| ![Screen 1][screen1] | ![Screen 2][screen2] | 
+|----------------------|----------------------| 
+| ![Screen 3][screen3] | ![Screen 4][screen4] | 
+
 PiAlert comes with a plugin system to feed events from third-party scripts into the UI and then send notifications, if desired. The highlighted functionality this plugin system supports, is dynamic creation of a simple UI to interact with the discovered objects, a mechanism to surface settings of plugins in the UI, or to import objects into existing PiAlert database tables. (Currently update/overwriting of existing objects is not supported.)
 
 Example use cases for plugins could be:
@@ -27,6 +19,23 @@ If you wish to develop a plugin, please check the existing plugin structure. Onc
 
 Again, please read the below carefully if you'd like to contribute with a plugin yourself. This documentation file might be outdated, so double check the sample plugins as well. 
 
+## ⚠ Disclaimer
+
+Experimental feature used also to speed up development  and to make the app more maintainable. Follow the below very carefully and check example plugin(s) if you'd like to write one yourself. Plugin UI is not my priority right now, happy to approve PRs if you are interested in extending/improvintg the UI experience. Example improvements for the taking:
+
+* Making the tables sortable/filterable
+* Using the same approach to display table  data as in the Devices section (solves above)
+* Adding form controls supported to display the data (Currently supported ones are listed in the section "UI settings in database_column_definitions" below)
+* ...
+
+## ❗ Known issues:
+
+These issues will be hopefully fixed with time, so please don't report them. Instead, if you know how, feel free to investigate and submit a PR to fix the below. Keep the PRs small as it's easier to approve them:
+
+* Existing plugin objects sometimes not interpreted correctly and a new object is created instead, resulting in duplicate entries.
+* Occasional (experienced twice) hanging of processing plugin script file.
+* UI displaying outdated values until the API endpoints get refreshed. 
+
 ## Plugin file structure overview 
 
 > Folder name must be the same as the code name value in: `"code_name": "<value>"`
@@ -86,7 +95,11 @@ Any of the above datasources have to return a "table" of the exact structure as
 - You can find which "columns" need to be present, and if the value is required or optional, in the "Column order and values" section. 
 - The order of these "columns" can't be changed.
 
-#### Examples
+### 👍 Python script.py tips
+
+The [Undicoverables plugins `script.py` file](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/undiscoverables/script.py) is a good and simple example to start with if you are considering creating a custom plugin. It uses the [`plugin_helper.py` library](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/plugin_helper.py) that significantly simplifies the creation of your custom script.
+
+#### last_result.log examples
 
 Valid CSV:
 
@@ -113,7 +126,9 @@ https://www.google.com|null|2023-01-02 15:56:30|200|0.7898|
 
 ### "data_source":  "pialert-db-query"
 
-If the datasource is set to `pialert-db-query` the `CMD` setting needs to contain a SQL query rendering the columns as defined in the "Column order and values" section above. The order of columns is important.
+If the datasource is set to `pialert-db-query` the `CMD` setting needs to contain a SQL query rendering the columns as defined in the "Column order and values" section above. The order of columns is important. 
+
+This SQL query is executed on the `pialert.db` SQLite database file. 
 
 #### Examples
 
@@ -351,15 +366,16 @@ Example:
 The UI will adjust how columns are displayed in the UI based on the definition of the `database_column_definitions` object. Thease are the supported form controls and related functionality:
 
 - Only columns with `"show": true` and also with at least an English translation will be shown in the UI.
-- Supported types: `label`, `text`, `threshold`, `replace`
+- Supported types: `label`, `text`, `threshold`, `replace`, `deviceip`, `devicemac`, `url`. Check for details below, how columns behave based on the type.
   - `label` makes a column display only
-  - `text` makes a column editable
+  - `text` makes a column editable and a save icon is displayed next to it.
   - See below for information on `threshold`, `replace`
 - The `options` property is used in conjunction with these types:
   - `threshold` - The `options` array contains objects from lowest `maximum` to highest with corresponding `hexColor` used for the value background color if it's less than the specified `maximum`, but more than the previous one in the `options` array
   - `replace` - The `options` array contains objects with an `equals` property, that is compared to the "value" and if the values are the same, the string in `replacement` is displayed in the UI instead of the actual "value"
-  - `devicemac` - The value is considered to be a mac adress and a link pointing to the device with the given mac address is generated.
-  - `url` - The value is considered to be a url so a link is generated.
+- `devicemac` - The value is considered to be a mac address and a link pointing to the device with the given mac address is generated.
+- `deviceip` - The value is considered to be an IP address and a link pointing to the device with the given IP is generated. The IP is cheked against the last detected IP addresses and translated into a mac address that is then used for the link itself.
+- `url` - The value is considered to be a url so a link is generated.
 
 
 ```json
@@ -431,16 +447,14 @@ The UI will adjust how columns are displayed in the UI based on the definition o
 - [website_monitor (WEBMON) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/website_monitor/config.json) 
 - [dhcp_servers (DHCPSRVS) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/dhcp_servers/config.json) 
 - [dhcp_leases (DHCPLSS) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/dhcp_leases/config.json) 
-- [unifi_import (UNFIMP) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/unifi_import/config.json) 
+- [unifi_import (UNFIMP) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/unifi_import/config.json)
+- [snmp_discovery (SNMPDSC) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/snmp_discovery/config.json)
+- [undiscoverables (UNDIS) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/undiscoverables/config.json)
 
 ### SQL query based plugins
 - [nmap_services (NMAPSERV) config.json](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/nmap_services/config.json) 
 
-### Screenshots 
 
-| ![Screen 1][screen1] | ![Screen 2][screen2] | 
-|----------------------|----------------------| 
-| ![Screen 3][screen3] | ![Screen 4][screen4] | 
 
 [screen1]: https://raw.githubusercontent.com/jokob-sk/Pi.Alert/main/docs/img/plugins.png                "Screen 1"
 [screen2]: https://raw.githubusercontent.com/jokob-sk/Pi.Alert/main/docs/img/plugins_settings.png       "Screen 2"

front/plugins/dhcp_leases/README.md

@@ -29,4 +29,25 @@ DHCPLSS_paths_to_check = ['/mnt/dhcp1.leases','/mnt/dhcp2.leases']
 
 ### Notes
 
-- No specific configuration needed.
+- No specific configuration needed.
+
+- This plugin expects the dhcp.leases file(s) to be in the format of **dhcpd.leases** that is different to the format that PiHole uses. 
+[dhcpd.leases(5) - Linux man page]( https://linux.die.net/man/5/dhcpd.leases#:~:text=This%20database%20is%20a%20free,file%20is%20the%20current%20one.) 
+
+Example File Format:  _(not all lines are required)_
+
+```
+lease 192.168.79.15 {
+  starts 0 2016/08/21 13:25:45;
+  ends 0 2016/08/21 19:25:45;
+  cltt 0 2016/08/21 13:25:45;
+  binding state active;
+  next binding state free;
+  rewind binding state free;
+  hardware ethernet 8c:1a:bf:11:00:ea;
+  uid "\001\214\032\277\021\000\352";
+  option agent.circuit-id 0:17;
+  option agent.remote-id c0:a8:9:5;
+  client-hostname "android-8182e21c852776e7";
+}  
+```

front/plugins/nmap_services/config.json

@@ -49,13 +49,13 @@
             "column": "Object_PrimaryID",
             "css_classes": "col-sm-2",
             "show": true,
-            "type": "label",            
+            "type": "devicemac",            
             "default_value":"",
             "options": [],
             "localized": ["name"],
             "name":[{
                 "language_code":"en_us",
-                "string" : "Device name"
+                "string" : "Device MAC"
                 }]
         },
         {
@@ -220,7 +220,7 @@
         {
             "function": "CMD",
             "type": "text",
-            "default_value":"SELECT  dv.dev_Name as Object_PrimaryID, cast('http://' || dv.dev_LastIP as VARCHAR(100)) || ':' || cast( SUBSTR(ns.Port ,0, INSTR(ns.Port , '/')) as VARCHAR(100)) as Object_SecondaryID,  datetime() as DateTime,  ns.Service as Watched_Value1,        ns.State as Watched_Value2,        'null' as Watched_Value3,        'null' as Watched_Value4,        ns.Extra as Extra, dv.dev_MAC as ForeignKey        FROM (SELECT * FROM Nmap_Scan) ns LEFT JOIN (SELECT dev_Name, dev_MAC, dev_LastIP FROM Devices) dv   ON ns.MAC = dv.dev_MAC",
+            "default_value":"SELECT  ns.MAC as Object_PrimaryID, cast('http://' || dv.dev_LastIP as VARCHAR(100)) || ':' || cast( SUBSTR(ns.Port ,0, INSTR(ns.Port , '/')) as VARCHAR(100)) as Object_SecondaryID,  datetime() as DateTime,  ns.Service as Watched_Value1, ns.State as Watched_Value2, dv.dev_Name as Watched_Value3,        'null' as Watched_Value4,        ns.Extra as Extra, ns.MAC as ForeignKey FROM (SELECT * FROM Nmap_Scan) ns left JOIN (SELECT dev_Name, dev_MAC, dev_LastIP FROM Devices) dv  ON ns.MAC = dv.dev_MAC",
             "options": [],
             "localized": ["name", "description"],
             "name" : [{

front/plugins/plugin_helper.py

@@ -0,0 +1,96 @@
+from time import strftime
+
+
+# -------------------------------------------------------------------
+class Plugin_Object:
+    """ 
+    Plugin_Object class to manage one object introduced by the plugin
+    An object typically is a device but could also be a website or something 
+    else that is monitored by the plugin.
+    """
+
+    def __init__(
+        self,
+        primaryId="",
+        secondaryId="",
+        watched1="",
+        watched2="",
+        watched3="",
+        watched4="",
+        extra="",
+        foreignKey="",
+    ):
+        self.pluginPref = ""
+        self.primaryId = primaryId
+        self.secondaryId = secondaryId
+        self.created = strftime("%Y-%m-%d %H:%M:%S")
+        self.changed = ""
+        self.watched1 = watched1
+        self.watched2 = watched2
+        self.watched3 = watched3
+        self.watched4 = watched4
+        self.status = ""
+        self.extra = extra
+        self.userData = ""
+        self.foreignKey = foreignKey
+
+    def write(self):
+        """ 
+        write the object details as a string in the 
+        format required to write the result file
+        """
+        line = "{}|{}|{}|{}|{}|{}|{}|{}|{}\n".format(
+            self.primaryId,
+            self.secondaryId,
+            self.created,
+            self.watched1,
+            self.watched2,
+            self.watched3,
+            self.watched4,
+            self.extra,
+            self.foreignKey,
+        )
+        return line
+
+
+class Plugin_Objects:
+    """
+    Plugin_Objects is the class that manages and holds all the objects created by the plugin.
+    It contains a list of Plugin_Object instances.
+    And can write the required result file.
+    """
+    
+    def __init__(self, result_file):
+        self.result_file = result_file
+        self.objects = []
+
+    def add_object(
+        self,
+        primaryId="",
+        secondaryId="",
+        watched1="",
+        watched2="",
+        watched3="",
+        watched4="",
+        extra="",
+        foreignKey="",
+    ):
+        self.objects.append(
+            Plugin_Object(
+                primaryId,
+                secondaryId,
+                watched1,
+                watched2,
+                watched3,
+                watched4,
+                extra,
+                foreignKey,
+            )
+        )
+
+    def write_result_file(self):
+        # print ("writing file: "+self.result_file)
+        with open(self.result_file, mode="w") as fp:
+            for obj in self.objects:
+                fp.write(obj.write())
+        fp.close()

front/plugins/snmp_discovery/README.md

@@ -0,0 +1,36 @@
+## Overview
+
+A plugin for importing devices from an SNMP enabled router or switch.  Using SNMP offers an efficient way to discover IPv4 devices across one or more networks/subnets/vlans.
+
+### Usage
+
+Specify the following settings in the Settings section of PiAlert:
+
+- `SNMPDSC_routers` - A list of `snmpwalk` commands to execute against IP addresses of roputers/switches with SNMP turned on. For example: `snmpwalk -v 2c -c public -OXsq 192.168.1.1 .1.3.6.1.2.1.3.1.1.2`
+
+
+### Setup Cisco IOS
+
+Enable IOS SNMP service and restrict to selected (internal) IP/Subnet.
+
+````
+! Add standard ip access-list 10
+ip access-list standard 10
+ permit 192.168.1.0 0.0.0.255
+ permit host 192.168.2.10
+!
+! Enable IOS snmp server with Read Only community 'mysnmpcommunitysecret' name.
+! Restrict connections to access-list 10
+snmp-server community mysnmpcommunitysecret RO 10
+````
+
+Confirm SNMP enabled
+````
+show snmp
+````
+
+### Notes
+
+- Only IPv4 supported.
+- The SNMP OID `.1.1.1.3.6.1.2.1.3.1.1.2` is specifically for devices IPv4 ARP table. This OID has been tested on Cisco ISRs and other L3 devices. Support may vary between other vendors / devices.
+- Expected output (ingestion) in format `iso.3.6.1.2.1.3.1.1.2.3.1.192.168.1.2 "6C 6C 6C 6C 6C 6C "`.

front/plugins/snmp_discovery/config.json

@@ -0,0 +1,328 @@
+{
+    "code_name": "snmp_discovery",
+    "unique_prefix": "SNMPDSC",
+    "enabled": true,    
+    "data_source":  "python-script",
+    "localized": ["display_name", "description", "icon"],
+    "mapped_to_table": "DHCP_Leases",    
+    "display_name" : [{
+        "language_code":"en_us",
+        "string" : "SNMP discovery"
+    }],
+    "icon":[{
+        "language_code":"en_us",
+        "string" : "<i class=\"fa-solid fa-s\"></i>"
+    }],     
+    "description": [{
+        "language_code":"en_us",
+        "string" : "This plugin is used to discover devices via the arp table(s) of a RFC1213 compliant router or switch."
+    }],
+    "params" : [
+        {
+            "name"  : "routers",
+            "type"  : "setting",
+            "value" : "SNMPDSC_routers"
+        }
+    ], 
+    "database_column_definitions":
+    [          
+        {
+            "column": "Index",            
+            "css_classes": "col-sm-2",
+            "show": false,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "N/A"
+                }]
+        } ,
+        {
+            "column": "Plugin",
+            "css_classes": "col-sm-2",
+            "show": false,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "N/A"
+                }]
+        },
+        {
+            "column": "Object_PrimaryID",
+            "mapped_to_column": "DHCP_MAC", 
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "devicemac",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "MAC address"
+                }]
+        },
+        {
+            "column": "Object_SecondaryID",
+            "mapped_to_column": "DHCP_IP", 
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "deviceip",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "IP"
+                }]
+        } ,
+        {
+            "column": "DateTimeCreated",
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Created"
+                }]
+        },
+        {
+            "column": "DateTimeChanged",
+            "mapped_to_column": "DHCP_DateTime", 
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Changed"
+                }]
+        },
+        {
+            "column": "Watched_Value1",
+            "mapped_to_column": "DHCP_Name", 
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "label",            
+            "default_value":"(unknown)",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Hostname"
+                }]
+        },
+        {
+            "column": "Watched_Value2",            
+            "css_classes": "col-sm-2",
+            "show": true,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Router IP"
+                }]
+        },
+        {
+            "column": "Watched_Value3",
+            "css_classes": "col-sm-2",
+            "show": false,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Type"
+                }]
+        } ,
+        {
+            "column": "Watched_Value4",
+            "css_classes": "col-sm-2",
+            "show": false,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Network"
+                }]
+        } ,       
+        {
+            "column": "UserData",
+            "css_classes": "col-sm-2",
+            "show": false,
+            "type": "textboxsave",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Comments"
+                }]
+        },        
+        {
+            "column": "Extra",
+            "css_classes": "col-sm-3",
+            "show": true,
+            "type": "label",            
+            "default_value":"",
+            "options": [],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "RAW output"
+                }]
+        },        
+        {
+            "column": "Status",
+            "css_classes": "col-sm-1",
+            "show": true,
+            "type": "replace",            
+            "default_value":"",
+            "options": [
+                {
+                    "equals": "watched-not-changed",
+                    "replacement": "<div style='text-align:center'><i class='fa-solid fa-square-check'></i><div></div>"
+                },
+                {
+                    "equals": "watched-changed",
+                    "replacement": "<div style='text-align:center'><i class='fa-solid fa-triangle-exclamation'></i></div>"
+                },
+                {
+                    "equals": "new",
+                    "replacement": "<div style='text-align:center'><i class='fa-solid fa-circle-plus'></i></div>"
+                }
+            ],
+            "localized": ["name"],
+            "name":[{
+                "language_code":"en_us",
+                "string" : "Status"
+                }]
+        }            
+    ],
+    "settings":[
+        {
+            "function": "RUN",            
+            "type": "selecttext",            
+            "default_value":"disabled",
+            "options": ["disabled", "once", "schedule", "always_after_scan", "on_new_device"],
+            "localized": ["name", "description"],
+            "name" :[{
+                "language_code":"en_us",
+                "string" : "When to run"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "Enable import of devices from a SNMP enabled device. If you select <code>schedule</code> the scheduling settings from below are applied. If you select <code>once</code> the scan is run only once on start of the application (container) or after you update your settings."
+            }]
+        },
+        {
+            "function": "CMD",
+            "type": "text",
+            "default_value":"python3 /home/pi/pialert/front/plugins/snmp_discovery/script.py routers={s-quote}{routers}{s-quote}",
+            "options": [],
+            "localized": ["name", "description"],
+            "name" : [{
+                "language_code":"en_us",
+                "string" : "Command"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "Command to run. Not recommended to change."
+            }]
+        },
+        {
+            "function": "routers",
+            "type": "list",
+            "default_value":["snmpwalk -v 2c -c public -OXsq 192.168.1.1 .1.3.6.1.2.1.3.1.1.2"],
+            "options": [],
+            "localized": ["name", "description"],
+            "name" : [{
+                "language_code":"en_us",
+                "string" : "Routers"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "A list of <code>snmpwalk</code> commands to execute against IP addresses of roputers/switches with SNMP turned on. <br/> <br/> Example with the router on the IP <code>192.168.1.1</code>: <br/> <code>snmpwalk -v 2c -c public -OXsq 192.168.1.1 .1.3.6.1.2.1.3.1.1.2</code> <br/><br/> Only IPv4 supported. Authentication is not supported. More info on the plugin <a href='https://github.com/jokob-sk/Pi.Alert/tree/main/front/plugins/snmp_discovery' target='_blank'>here</a>."
+            }]
+        },
+        {
+            "function": "RUN_SCHD",
+            "type": "text",
+            "default_value":"0 2 * * *",
+            "options": [],
+            "localized": ["name", "description"],
+            "name" : [{
+                "language_code":"en_us",
+                "string" : "Schedule"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#SNMPDSC_RUN\"><code>SNMPDSC_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>0 4 * * *</code> will run the scan after 4 am in the <a onclick=\"toggleAllSettings()\" href=\"#TIMEZONE\"><code>TIMEZONE</code> you set above</a>. Will be run NEXT time the time passes."
+            }]
+        },
+        {
+            "function": "RUN_TIMEOUT",
+            "type": "integer",
+            "default_value":5,
+            "options": [],
+            "localized": ["name", "description"],
+            "name" : [{
+                "language_code":"en_us",
+                "string" : "Run timeout"
+            },
+            {
+                "language_code":"de_de",
+                "string" : "Wartezeit"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "Maximum time in seconds to wait for the script to finish. If this time is exceeded the script is aborted."
+            }]
+        },
+        {
+            "function": "WATCH",
+            "type": "multiselect",
+            "default_value":["Watched_Value1"],
+            "options": ["Watched_Value1","Watched_Value2","Watched_Value3","Watched_Value4"],
+            "localized": ["name", "description"],
+            "name" :[{
+                "language_code":"en_us",
+                "string" : "Watched"
+            }] ,
+            "description":[{
+                "language_code":"en_us",
+                "string" : "Send a notification if selected values change. Use <code>CTRL + Click</code> to select/deselect. <ul> <li><code>Watched_Value1</code> is Hostname (not discoverable) </li><li><code>Watched_Value2</code> is Router IP </li><li><code>Watched_Value3</code> is not used </li><li><code>Watched_Value4</code> is not used </li></ul>"
+            }] 
+        },
+        {
+            "function": "REPORT_ON",
+            "type": "multiselect",
+            "default_value":["new","watched-changed"],
+            "options": ["new","watched-changed","watched-not-changed"],
+            "localized": ["name", "description"],
+            "name" :[{
+                "language_code":"en_us",
+                "string" : "Report on"
+            }] ,
+            "description":[{
+                "language_code":"en_us",
+                "string" : "Send a notification only on these statuses. <code>new</code> means a new unique (unique combination of PrimaryId and SecondaryId) object was discovered. <code>watched-changed</code> means that selected <code>Watched_ValueN</code> columns changed."
+            }] 
+        }
+    ]
+}
+

front/plugins/snmp_discovery/script.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python
+
+# Example call
+# python3 /home/pi/pialert/front/plugins/snmp_discovery/script.py routers='snmpwalk -v 2c -c public -OXsq 192.168.1.1 .1.3.6.1.2.1.3.1.1.2'
+
+from __future__ import unicode_literals
+from time import sleep, time, strftime
+import requests
+from requests                               import Request, Session, packages
+import pathlib
+import threading
+import subprocess
+import socket
+import json
+import argparse
+import io
+import sys
+from requests.packages.urllib3.exceptions import InsecureRequestWarning
+import pwd
+import os
+
+
+curPath = str(pathlib.Path(__file__).parent.resolve())
+log_file = curPath + '/script.log'
+last_run = curPath + '/last_result.log'
+
+# Workflow
+
+def main():    
+
+    # init global variables
+    global ROUTERS
+
+    # empty file
+    open(last_run , 'w').close()   
+
+    last_run_logfile = open(last_run, 'a') 
+    
+
+    parser = argparse.ArgumentParser(description='This plugin is used to discover devices via the arp table(s) of a RFC1213 compliant router or switch.')
+    
+    parser.add_argument('routers',  action="store",  help="IP(s) of routers, separated by comma (,) if passing multiple")                        
+
+    values = parser.parse_args()
+
+    # parse output
+    newEntries = []
+
+    if values.routers:        
+
+        ROUTERS = values.routers.split('=')[1].replace('\'','')  
+        
+        newEntries = get_entries(newEntries)  
+   
+    for e in newEntries:        
+        # Insert list into the log            
+        service_monitoring_log(e.primaryId, e.secondaryId, e.created, e.watched1, e.watched2, e.watched3, e.watched4, e.extra, e.foreignKey )
+
+# -----------------------------------------------------------------------------
+def get_entries(newEntries):
+
+    routers = []
+
+    if ',' in ROUTERS:
+        # multiple
+        routers = ROUTERS.split(',')
+    
+    else:
+        # only one
+        routers.append(ROUTERS)
+
+    for router in routers:
+        # snmpwalk -v 2c -c public -OXsq 192.168.1.1 .1.3.6.1.2.1.3.1.1.2
+
+        print(router)
+
+        timeoutSec = 10
+
+        snmpwalkArgs = router.split(' ')
+
+        # Execute N probes and insert in list
+        probes = 1 #  N probes
+        newLines = []                
+        for _ in range(probes):
+            output = subprocess.check_output (snmpwalkArgs, universal_newlines=True,  stderr=subprocess.STDOUT, timeout=(timeoutSec ))
+            newLines = newLines + output.split("\n")
+
+        # Process outputs
+        # Sample: iso.3.6.1.2.1.3.1.1.2.3.1.192.168.1.2 "6C 6C 6C 6C 6C 6C "
+
+        with open(log_file, 'a') as run_logfile:
+            for line in newLines:              
+                # debug
+                run_logfile.write(line)
+
+                tmpSplt = line.split('"')   
+
+                if len(tmpSplt) == 3:
+
+                    ipStr   = tmpSplt[0].split('.') # contains IP
+
+                    macStr  = tmpSplt[1].split(' ') # contains MAC
+
+                    if 'iso.' in line and len(ipStr) == 16:
+                        tmpEntry = plugin_object_class(
+                            f'{macStr[0]}:{macStr[1]}:{macStr[2]}:{macStr[3]}:{macStr[4]}:{macStr[5]}',
+                            f'{ipStr[12]}.{ipStr[13]}.{ipStr[14]}.{ipStr[15]}'.strip(),
+                            watched1='(unknown)',
+                            watched2=snmpwalkArgs[6], # router IP
+                            extra=line
+                        )
+                        newEntries.append(tmpEntry) 
+        
+    return newEntries
+
+
+# -------------------------------------------------------------------
+class plugin_object_class:
+    def __init__(self, primaryId = '',secondaryId = '', watched1 = '',watched2 = '',watched3 = '',watched4 = '',extra = '',foreignKey = ''):        
+        self.pluginPref   = ''
+        self.primaryId    = primaryId
+        self.secondaryId  = secondaryId
+        self.created      = strftime("%Y-%m-%d %H:%M:%S")
+        self.changed      = ''
+        self.watched1     = watched1
+        self.watched2     = watched2
+        self.watched3     = watched3
+        self.watched4     = watched4
+        self.status       = ''
+        self.extra        = extra
+        self.userData     = ''
+        self.foreignKey   = foreignKey
+
+# -----------------------------------------------------------------------------
+def service_monitoring_log(primaryId, secondaryId, created, watched1, watched2 = 'null', watched3 = 'null', watched4 = 'null', extra ='null', foreignKey ='null'  ):
+    
+    if watched1 == '':
+        watched1 = 'null'
+    if watched2 == '':
+        watched2 = 'null'
+    if watched3 == '':
+        watched3 = 'null'
+    if watched4 == '':
+        watched4 = 'null'
+    if extra == '':
+        extra = 'null'
+    if foreignKey == '':
+        foreignKey = 'null'
+
+    with open(last_run, 'a') as last_run_logfile:        
+        last_run_logfile.write("{}|{}|{}|{}|{}|{}|{}|{}|{}\n".format(
+                                                primaryId,
+                                                secondaryId,
+                                                created,                                                
+                                                watched1,
+                                                watched2,
+                                                watched3,
+                                                watched4,
+                                                extra,
+                                                foreignKey
+                                                )
+                             )
+
+
+#===============================================================================
+# BEGIN
+#===============================================================================
+if __name__ == '__main__':    
+    main()  
+

front/plugins/undiscoverables/README.md

@@ -0,0 +1,27 @@
+## Overview
+
+A plugin allowing for importing Un-Discoverable devices from the settings page.
+The main usecase is to add dumb network gear like unmanaged hubs and switches to the network view.
+There might be other usecases, please let me know.
+
+### Usage
+
+- Go to settings and find Un-Discoverabe Devices in the list of plugins.
+- Enable the plugin by changing the RUN parameter from disabled to `once` or `always_after_scan`.
+- Add the name of your device to the list. (remove the sample entry first)
+- SAVE
+- wait for the next scan to finish
+
+#### Examples:
+Settings:
+![settings](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/52883307-19a5-4602-b13a-9825461f6cc4)
+
+resulting in these devices:
+![devices](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/9f7659e7-75a8-4ae9-9f5f-781bdbcbc949)
+
+Allowing Un-Discoverable devices like hubs, switches or APs to be added to the network view.
+![network](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/b5ccc3b3-f5fd-4f5b-b0f0-e4e637c6da33)
+
+### Known Limitations
+ - Un-Discoverable Devices always show as offline. That is expected as they can not be discovered by Pi.Alert.
+ - All IPs are set to 0.0.0.0 therefore the "Random MAC" icon might show up.

front/plugins/undiscoverables/config.json

@@ -0,0 +1,217 @@
+{
+  "code_name": "undiscoverables",
+  "unique_prefix": "UNDIS",
+  "enabled": true,
+  "data_source": "python-script",
+  "mapped_to_table": "DHCP_Leases", 
+
+  "localized": ["display_name", "description", "icon"],
+
+  "display_name": [
+    {
+      "language_code": "en_us",
+      "string": "Un-Discoverable Devices"
+    }
+  ],
+  "icon": [
+    {
+      "language_code": "en_us",
+      "string": "<i class=\"fa-solid fa-binoculars\"></i>"
+    }
+  ],
+  "description": [
+    {
+      "language_code": "en_us",
+      "string": "This plugin is to import undiscoverable devices from a file."
+    }
+  ],
+    "params" : [
+    {
+        "name"  : "devices",
+        "type"  : "setting",
+        "value" : "UNDIS_devices_to_import"
+    }],  
+
+  "settings": [
+    {
+      "function": "RUN",            
+      "type": "selecttext",            
+      "default_value":"disabled",
+      "options": ["disabled", "once", "always_after_scan"],
+      "localized": ["name", "description"],
+      "name" :[{
+          "language_code":"en_us",
+          "string" : "When to run"
+      }],
+      "description": [{
+          "language_code":"en_us",
+          "string" : "When enabled, ONCE is the preferred option. It runs at startup and after every save of the config here.<br> Changes will only show in the devices <b> after the next scan!</b>"
+      }]
+  },    
+    {
+      "function": "CMD",
+      "type": "readonly",
+      "default_value": "python3 /home/pi/pialert/front/plugins/undiscoverables/script.py devices={devices}",
+      "options": [],
+      "localized": ["name", "description"],
+      "name": [
+        {
+          "language_code": "en_us",
+          "string": "Command"
+        }
+      ],
+      "description": [
+        {
+          "language_code": "en_us",
+          "string": "Command to run. This can not be changed"
+        }
+      ]
+    },
+    
+    {
+      "function": "RUN_TIMEOUT",
+      "type": "integer",
+      "default_value": 10,
+      "options": [],
+      "localized": ["name", "description"],
+      "name": [
+        {
+          "language_code": "en_us",
+          "string": "Run timeout"
+        }
+      ],
+      "description": [
+        {
+          "language_code": "en_us",
+          "string": "Maximum time in seconds to wait for the script to finish. If this time is exceeded the script is aborted."
+        }
+      ]
+    },
+    {
+      "function": "WATCH",
+      "type": "lable",
+      "default_value": [],
+      "options": [],
+      "localized": ["name", "description"],
+      "name": [
+        {
+          "language_code": "en_us",
+          "string": "Watched"
+        }
+      ],
+      "description": [
+        {
+          "language_code": "en_us",
+          "string": "Undiscoverable Devices can not change their status, no watch is enabled."
+        }
+      ]
+    },
+    {
+      "function": "REPORT_ON",
+      "type": "lable",
+      "default_value": [],
+      "options": ["new", "watched-changed", "watched-not-changed"],
+      "localized": ["name", "description"],
+      "name": [
+        {
+          "language_code": "en_us",
+          "string": "Report on"
+        }
+      ],
+      "description": [
+        {
+          "language_code": "en_us",
+          "string": "No notifications will be sent."
+        }
+      ]
+    },
+    {
+      "function": "devices_to_import",
+      "type": "list",
+      "default_value":["dummy_router"],
+      "options": [],
+      "localized": ["name", "description"],
+      "name" : [{
+          "language_code":"en_us",
+          "string" : "UnDiscoverable Devices"
+      }],
+      "description": [{
+          "language_code":"en_us",
+          "string" : "Devices to be added to the devices list."
+      }]
+  }
+  ],
+
+  "database_column_definitions": 
+  [          
+    {
+      "column": "Watched_Value1",
+      "mapped_to_column": "DHCP_Name",         
+      "css_classes": "col-sm-2",
+      "show": true,
+      "type": "label",            
+      "default_value":"",
+      "options": [],
+      "localized": ["name"],
+      "name":[{
+          "language_code":"en_us",
+          "string" : "Device Name"
+          }]
+    }, 
+    {
+        "column": "Object_PrimaryID",
+        "mapped_to_column": "DHCP_MAC", 
+        "css_classes": "col-sm-2",
+        "show": true,
+        "type": "devicemac",            
+        "default_value":"",
+        "options": [],
+        "localized": ["name"],
+        "name":[{
+            "language_code":"en_us",
+            "string" : "MAC address"
+            }]
+    },
+    {
+        "column": "Object_SecondaryID",
+        "mapped_to_column": "DHCP_IP", 
+        "css_classes": "col-sm-2",
+        "show": true,
+        "type": "deviceip",            
+        "default_value":"",
+        "options": [],
+        "localized": ["name"],
+        "name":[{
+            "language_code":"en_us",
+            "string" : "IP"
+            }]
+    } ,
+    {
+      "column": "DateTimeCreated",
+      "css_classes": "col-sm-2",
+      "show": true,
+      "type": "label",            
+      "default_value":"",
+      "options": [],
+      "localized": ["name"],
+      "name":[{
+          "language_code":"en_us",
+          "string" : "Created"
+          }]
+    },   
+    {
+        "column": "DateTimeChanged",
+        "mapped_to_column": "DHCP_DateTime", 
+        "css_classes": "col-sm-2",
+        "show": true,
+        "type": "label",            
+        "default_value":"",
+        "options": [],
+        "localized": ["name"],
+        "name":[{
+            "language_code":"en_us",
+            "string" : "Changed"
+            }]
+    }        
+  ]
+}

front/plugins/undiscoverables/script.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+# test script by running python script.py devices=test,dummy
+
+import os
+import pathlib
+import argparse
+import sys
+
+sys.path.append("/home/pi/pialert/front/plugins")
+
+from plugin_helper import Plugin_Objects
+
+CUR_PATH = str(pathlib.Path(__file__).parent.resolve())
+LOG_FILE = os.path.join(CUR_PATH , 'script.log')
+RESULT_FILE = os.path.join(CUR_PATH , 'last_result.log')
+
+
+
+def main():
+
+    # the script expects a parameter in the format of devices=device1,device2,...
+    parser = argparse.ArgumentParser(description='Import devices from settings')
+    parser.add_argument('devices',  action="store",  help="list of device names separated by ','")
+    values = parser.parse_args()
+
+    UNDIS_devices = Plugin_Objects( RESULT_FILE )
+
+    if values.devices:
+        for fake_dev in values.devices.split('=')[1].split(','):
+            UNDIS_devices.add_object(
+                primaryId=fake_dev,    # MAC (Device Name)
+                secondaryId="0.0.0.0", # IP Address (always 0.0.0.0)
+                watched1=fake_dev,     # Device Name
+                watched2="",
+                watched3="",
+                watched4="",
+                extra="",
+                foreignKey="")
+
+    UNDIS_devices.write_result_file()
+
+    return 0
+
+#===============================================================================
+# BEGIN
+#===============================================================================
+if __name__ == '__main__':
+    main()

front/plugins/unifi_import/README.md

@@ -4,7 +4,7 @@ A plugin allowing for importing devices from a UniFi controller.
 
 ### Usage
 
-Spedify the following settings in the Settings section of PiAlert:
+Specify the following settings in the Settings section of PiAlert:
 
 - `UNFIMP_username` - Username used to login into the UNIFI controller.
 - `UNFIMP_password` - Password used to login into the UNIFI controller.
@@ -15,4 +15,5 @@ Spedify the following settings in the Settings section of PiAlert:
 
 ### Notes
 
-- Currently only used to import devices, not their status, type or network map.
+- Currently only used to import devices, not their status, type or network map.
+- It is recommend to create a read-only user in your UniFi controller 

front/plugins/unifi_import/config.json

@@ -47,6 +47,11 @@
             "name"  : "port",
             "type"  : "setting",
             "value" : "UNFIMP_port"
+        },
+        {
+            "name"  : "version",
+            "type"  : "setting",
+            "value" : "UNFIMP_version"
         }
     ], 
     "database_column_definitions":
@@ -182,7 +187,7 @@
             "localized": ["name"],
             "name":[{
                 "language_code":"en_us",
-                "string" : "Network"
+                "string" : "Is online?"
                 }]
         } ,       
         {
@@ -208,7 +213,7 @@
             "localized": ["name"],
             "name":[{
                 "language_code":"en_us",
-                "string" : "Hostname"
+                "string" : "Network"
                 }]
         },        
         {
@@ -257,7 +262,7 @@
         {
             "function": "CMD",
             "type": "text",
-            "default_value":"python3 /home/pi/pialert/front/plugins/unifi_import/script.py username={username} password={password}  host={host} sites={sites}  protocol={protocol} port={port}",
+            "default_value":"python3 /home/pi/pialert/front/plugins/unifi_import/script.py username={username} password={password}  host={host} sites={sites}  protocol={protocol} port={port} version={version}",
             "options": [],
             "localized": ["name", "description"],
             "name" : [{
@@ -344,6 +349,21 @@
                 "string" : "The port number where the UNIFI controller is runnig. Usually it is <code>8443</code>."
             }]
         },
+        {
+            "function": "version",
+            "type": "text",
+            "default_value":"",
+            "options": [],
+            "localized": ["name", "description"],
+            "name" : [{
+                "language_code":"en_us",
+                "string" : "API version"
+            }],
+            "description": [{
+                "language_code":"en_us",
+                "string" : "The base version of the Unify controller API. Supported values as of time of writing are <code>v4|v5|unifiOS|UDMP-unifiOS</code>."
+            }]
+        },
         {
             "function": "sites",
             "type": "list",
@@ -371,7 +391,7 @@
             }],
             "description": [{
                 "language_code":"en_us",
-                "string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#DHCPLSS_RUN\"><code>DHCPLSS_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>0 4 * * *</code> will run the scan after 4 am in the <a onclick=\"toggleAllSettings()\" href=\"#TIMEZONE\"><code>TIMEZONE</code> you set above</a>. Will be run NEXT time the time passes."
+                "string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#UNFIMP_RUN\"><code>UNFIMP_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>0 4 * * *</code> will run the scan after 4 am in the <a onclick=\"toggleAllSettings()\" href=\"#TIMEZONE\"><code>TIMEZONE</code> you set above</a>. Will be run NEXT time the time passes."
             }]
         },
         {

front/plugins/unifi_import/script.py

@@ -32,7 +32,7 @@ def main():
 
     # init global variables
     global UNIFI_USERNAME, UNIFI_PASSWORD, UNIFI_HOST
-    global UNIFI_SITES, PORT, PROTOCOL
+    global UNIFI_SITES, PORT, PROTOCOL, VERSION
 
     last_run_logfile = open(last_run, 'a') 
 
@@ -47,6 +47,7 @@ def main():
     parser.add_argument('sites',  action="store",  help="Name of the sites (usually 'default', check the URL in your UniFi controller UI). Separated by comma (,) if passing multiple sites")      
     parser.add_argument('protocol',  action="store",  help="https:// or http://")  
     parser.add_argument('port',  action="store",  help="Usually 8443")  
+    parser.add_argument('version',  action="store",  help="The base version of the controller API [v4|v5|unifiOS|UDMP-unifiOS]")  
 
     values = parser.parse_args()
 
@@ -61,6 +62,7 @@ def main():
         UNIFI_SITES = values.sites.split('=')[1]  
         PROTOCOL = values.protocol.split('=')[1]
         PORT = values.port.split('=')[1]
+        VERSION = values.version.split('=')[1]
         
         newEntries = get_entries(newEntries)  
 
@@ -85,7 +87,7 @@ def get_entries(newEntries):
 
     for site in sites:
 
-        c = Controller(UNIFI_HOST, UNIFI_USERNAME, UNIFI_PASSWORD, ssl_verify=False, site_id=site )
+        c = Controller(UNIFI_HOST, UNIFI_USERNAME, UNIFI_PASSWORD, version=VERSION, ssl_verify=False, site_id=site )
 
         for ap in c.get_aps():
 

front/plugins/website_monitor/script.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # Based on the work of https://github.com/leiweibau/Pi.Alert
 
-# /home/pi/pialert/front/plugins/website_monitor/script.py urls=http://google.com,http://bing.com
+# python3 /home/pi/pialert/front/plugins/website_monitor/script.py urls=http://google.com,http://bing.com
 from __future__ import unicode_literals
 from time import sleep, time, strftime
 import requests

front/presence.php

@@ -234,7 +234,7 @@ function initializeCalendar () {
     header: {
       left            : 'prev,next today',
       center          : 'title',
-      right           : 'timelineYear,timelineMonth,timelineWeek'
+      right           : 'timelineYear,timelineMonth,timelineWeek,timelineDay'
     },
     defaultView       : 'timelineMonth',
     height            : 'auto',
@@ -286,6 +286,13 @@ function initializeCalendar () {
         buttonText        : '<?= lang('Presence_CalHead_week');?>',
         slotLabelFormat   : 'D',
         slotDuration      : '24:00:01'
+      },
+      timelineDay: {
+        type              : 'timeline',
+        duration          : { day: 1 },
+        buttonText        : '<?= lang('Presence_CalHead_day');?>',
+        slotLabelFormat   : 'H',
+        slotDuration      : '00:30:00'
       }
     },
      
@@ -305,6 +312,15 @@ function initializeCalendar () {
 
       if (date.format('YYYY-MM-DD') == moment().format('YYYY-MM-DD')) {
           cell.addClass ('fc-today'); };
+          
+      if ($('#calendar').fullCalendar('getView').name == 'timelineDay') {
+        cell.removeClass('fc-sat');
+        cell.removeClass('fc-sun');
+        cell.removeClass('fc-today');
+        if (date.format('YYYY-MM-DD HH') == moment().format('YYYY-MM-DD HH')) {
+          cell.addClass('fc-today');
+        }
+      };
     },
     
     resourceRender: function (resourceObj, labelTds, bodyTds) {

pialert/README.md

@@ -0,0 +1,14 @@
+# Pi.Alert all split into modules
+
+I am trying to split this big original file into modules and gives me some nice challanges to solve.
+Since the original code is all in one file, the original author has taken quite some shortcuts by defining lots of variables as global !!
+These need to be changed now.
+
+Here is the main structure
+
+| Module | Description |
+|--------|-----------|
+|pialert.py | The MAIN program of Pi.Alert|
+|const.py | A place to define the constants for Pi.Alert like log path or config path.|
+|const.py| const.py holds the configuration variables and makes them availabe for all modules. It is also the <b>workaround</b> for the global variables until I can work them out|
+|api.py| |

pialert/__init__.py

@@ -0,0 +1 @@
+""" __init__ for Pi.Alert """

pialert/__main__.py

@@ -0,0 +1,314 @@
+#!/usr/bin/env python
+#
+#-------------------------------------------------------------------------------
+#  Pi.Alert  v2.70  /  2021-02-01
+#  Open Source Network Guard / WIFI & LAN intrusion detector 
+#
+#  pialert.py - Back module. Network scanner
+#-------------------------------------------------------------------------------
+#  Puche 2021 / 2022+ jokob             jokob@duck.com                GNU GPLv3
+#-------------------------------------------------------------------------------
+
+
+#===============================================================================
+# IMPORTS
+#===============================================================================
+#from __future__ import print_function
+
+import sys
+import time
+import datetime
+import multiprocessing
+
+# pialert modules
+import conf
+from const import *
+from logger import  mylog
+from helper import   filePermissions, isNewVersion,  timeNow, updateState
+from api import update_api
+from networkscan import process_scan, scan_network
+from initialise import importConfigs
+from mac_vendor import update_devices_MAC_vendors
+from database import DB, get_all_devices
+from reporting import check_and_run_event, send_notifications
+from plugin import run_plugin_scripts 
+
+# different scanners
+from scanners.pholusscan import performPholusScan
+from scanners.nmapscan import performNmapScan
+from scanners.internet import check_internet_IP
+
+
+
+#===============================================================================
+#===============================================================================
+#                              MAIN
+#===============================================================================
+#===============================================================================
+"""
+main structure of Pi Alert
+
+    Initialise All
+    start Loop forever
+        initialise loop 
+            (re)import config
+            (re)import plugin config
+        run plugins (once)
+        run frontend events
+        update API 
+        run scans
+            run plugins (scheduled)
+            check internet IP
+            check vendor
+            run PHOLUS
+            run NMAP
+            run "scan_network()"
+                ARP Scan
+                PiHole copy db
+                PiHole DHCP leases
+                processing scan results
+            run plugins (after Scan)
+        reporting
+        cleanup
+    end loop
+"""
+
+def main ():
+    mylog('debug', ['[MAIN] Setting up ...'])
+
+    conf.time_started = datetime.datetime.now()
+    conf.cycle = ""
+    conf.check_report = [1, "internet_IP", "update_vendors_silent"]
+    conf.plugins_once_run = False
+    
+    pialert_start_time = timeNow()
+
+
+    # to be deleted if not used 
+    conf.log_timestamp = conf.time_started
+    #cron_instance = Cron()
+
+    # timestamps of last execution times
+    startTime = conf.time_started
+    now_minus_24h = conf.time_started - datetime.timedelta(hours = 24)
+
+    # set these times to the past to force the first run 
+    last_network_scan = now_minus_24h
+    last_internet_IP_scan = now_minus_24h
+    last_scan_run = now_minus_24h
+    last_cleanup = now_minus_24h
+    last_update_vendors = conf.time_started - datetime.timedelta(days = 6) # update vendors 24h after first run and then once a week
+    last_version_check = now_minus_24h  
+
+    # indicates, if a new version is available
+    conf.newVersionAvailable = False
+    
+    # check file permissions and fix if required
+    filePermissions()
+
+    # Open DB once and keep open
+    # Opening / closing DB frequently actually casues more issues
+    db = DB()  # instance of class DB
+    db.open()
+    sql = db.sql  # To-Do replace with the db class
+
+    # Upgrade DB if needed
+    db.upgradeDB()
+
+
+    #===============================================================================
+    # This is the main loop of Pi.Alert 
+    #===============================================================================
+    while True:
+
+        # update time started
+        time_started = datetime.datetime.now()  # not sure why we need this ...
+        loop_start_time = timeNow()
+        mylog('debug', '[MAIN] Starting loop')
+
+        # re-load user configuration and plugins   
+        importConfigs(db)
+
+        # check if new version is available / only check once an hour
+        # if newVersionAvailable is already true the function does nothing and returns true again
+        if last_version_check  + datetime.timedelta(hours=1) < loop_start_time :
+            conf.newVersionAvailable = isNewVersion(conf.newVersionAvailable)
+
+        # Handle plugins executed ONCE
+        if conf.ENABLE_PLUGINS and conf.plugins_once_run == False:
+            run_plugin_scripts(db, 'once')  
+            conf.plugins_once_run = True
+
+        # check if there is a front end initiated event which needs to be executed
+        check_and_run_event(db)
+
+        # Update API endpoints              
+        update_api(db)
+        
+        # proceed if 1 minute passed
+        if last_scan_run + datetime.timedelta(minutes=1) < loop_start_time :
+
+             # last time any scan or maintenance/upkeep was run
+            last_scan_run = time_started
+
+            # Header
+            updateState(db,"Process: Start")      
+
+            # Timestamp
+            startTime = time_started
+            startTime = startTime.replace (microsecond=0) 
+
+            # Check if any plugins need to run on schedule
+            if conf.ENABLE_PLUGINS:
+                run_plugin_scripts(db,'schedule') 
+
+            # determine run/scan type based on passed time
+            # --------------------------------------------
+
+            # check for changes in Internet IP
+            if last_internet_IP_scan + datetime.timedelta(minutes=3) < time_started:
+                conf.cycle = 'internet_IP'                
+                last_internet_IP_scan = time_started
+                check_internet_IP(db)
+
+            # Update vendors once a week
+            if last_update_vendors + datetime.timedelta(days = 7) < time_started:
+                last_update_vendors = time_started
+                conf.cycle = 'update_vendors'
+                mylog('verbose', ['[MAIN] cycle:',conf.cycle])                  
+                update_devices_MAC_vendors(db)
+
+            # Execute scheduled or one-off Pholus scan if enabled and run conditions fulfilled
+            if conf.PHOLUS_RUN == "schedule" or conf.PHOLUS_RUN == "once":
+
+                pholusSchedule = [sch for sch in conf.mySchedules if sch.service == "pholus"][0]
+                run = False
+
+                # run once after application starts
+
+
+                if conf.PHOLUS_RUN == "once" and pholusSchedule.last_run == 0:
+                    run = True
+
+                # run if overdue scheduled time
+                if conf.PHOLUS_RUN == "schedule":
+                    run = pholusSchedule.runScheduleCheck()                    
+
+                if run:
+                    pholusSchedule.last_run = datetime.datetime.now(conf.tz).replace(microsecond=0)
+                    performPholusScan(db, conf.PHOLUS_RUN_TIMEOUT, conf.userSubnets)
+            
+            # Execute scheduled or one-off Nmap scan if enabled and run conditions fulfilled
+            if conf.NMAP_RUN == "schedule" or conf.NMAP_RUN == "once":
+
+                nmapSchedule = [sch for sch in conf.mySchedules if sch.service == "nmap"][0]
+                run = False
+
+                # run once after application starts
+                if conf.NMAP_RUN == "once" and nmapSchedule.last_run == 0:
+                    run = True
+
+                # run if overdue scheduled time
+                if conf.NMAP_RUN == "schedule":
+                    run = nmapSchedule.runScheduleCheck()                    
+
+                if run:
+                    nmapSchedule.last_run = timeNow()
+                    performNmapScan(db, get_all_devices(db))
+            
+            # Perform a network scan via arp-scan or pihole
+            if last_network_scan + datetime.timedelta(minutes=conf.SCAN_CYCLE_MINUTES) < time_started:
+                last_network_scan = time_started
+                conf.cycle = 1 # network scan
+                mylog('verbose', ['[MAIN] cycle:',conf.cycle])
+                updateState(db,"Scan: Network")
+
+                # scan_network() 
+
+                #  DEBUG start ++++++++++++++++++++++++++++++++++++++++++++++++++++++
+                # Start scan_network as a process                
+
+                p = multiprocessing.Process(target=scan_network(db))
+                p.start()
+
+                # Wait for 3600 seconds (max 1h) or until process finishes
+                p.join(3600)
+
+                # If thread is still active
+                if p.is_alive():
+                    mylog('none', "[MAIN]  scan_network running too long - let\'s kill it")
+
+                    # Terminate - may not work if process is stuck for good
+                    p.terminate()
+                    # OR Kill - will work for sure, no chance for process to finish nicely however
+                    # p.kill()
+
+                    p.join()
+
+                #  DEBUG end ++++++++++++++++++++++++++++++++++++++++++++++++++++++
+                # Run splugin scripts which are set to run every timne after a scan finished
+                if conf.ENABLE_PLUGINS:
+                    run_plugin_scripts(db,'always_after_scan')
+
+            # --------------------------------------------------
+            # process all the scanned data into new devices
+            mylog('debug', "[MAIN] start processig scan results")
+            process_scan (db, conf.arpscan_devices )
+            
+            # Reporting   
+            if conf.cycle in conf.check_report:
+                # Check if new devices found
+                sql.execute (sql_new_devices)
+                newDevices = sql.fetchall()
+                db.commitDB()
+                
+                #  new devices were found
+                if len(newDevices) > 0:
+                    #  run all plugins registered to be run when new devices are found
+                    if conf.ENABLE_PLUGINS:
+                        run_plugin_scripts(db, 'on_new_device')
+
+                    #  Scan newly found devices with Nmap if enabled
+                    if conf.NMAP_ACTIVE and len(newDevices) > 0:
+                        performNmapScan( db, newDevices)
+
+                # send all configured notifications
+                send_notifications(db)
+
+            # clean up the DB once a day
+            if last_cleanup + datetime.timedelta(hours = 24) < time_started:
+                last_cleanup = time_started
+                conf.cycle = 'cleanup'  
+                mylog('verbose', ['[MAIN] cycle:',conf.cycle])
+                db.cleanup_database(startTime, conf.DAYS_TO_KEEP_EVENTS, conf.PHOLUS_DAYS_DATA)   
+
+            # Commit SQL
+            db.commitDB()          
+            
+            # Final message
+            if conf.cycle != "":
+                action = str(conf.cycle)
+                if action == "1":
+                    action = "network_scan"
+                mylog('verbose', ['[MAIN] Last action: ', action])
+                conf.cycle = ""
+                mylog('verbose', ['[MAIN] cycle:',conf.cycle])
+            
+            # Footer
+            updateState(db,"Process: Wait")
+            mylog('verbose', ['[MAIN] Process: Wait'])            
+        else:
+            # do something
+            conf.cycle = "" 
+            mylog('verbose', ['[MAIN] waiting to start next loop'])
+
+        #loop     
+        time.sleep(5) # wait for N seconds      
+
+
+#===============================================================================
+# BEGIN
+#===============================================================================
+if __name__ == '__main__':
+    mylog('debug', ['[__main__] Welcome to Pi.Alert'])
+    sys.exit(main())       

pialert/api.py

@@ -0,0 +1,96 @@
+import json
+
+
+# pialert modules
+import conf  
+from const import (apiPath, sql_devices_all, sql_nmap_scan_all, sql_pholus_scan_all, sql_events_pending_alert, 
+                   sql_settings, sql_plugins_events, sql_plugins_history, sql_plugins_objects,sql_language_strings)
+from logger import mylog
+from helper import write_file
+
+apiEndpoints = []
+
+#===============================================================================
+# API
+#===============================================================================
+def update_api(db, isNotification = False, updateOnlyDataSources = []):
+    mylog('verbose', ['[API] Update API starting'])
+    # return
+
+    folder = apiPath 
+
+    # update notifications moved to reporting send_api()
+
+    # Save plugins
+    if conf.ENABLE_PLUGINS: 
+        write_file(folder + 'plugins.json'  , json.dumps({"data" : conf.plugins}))  
+
+    #  prepare database tables we want to expose 
+    dataSourcesSQLs = [
+        ["devices", sql_devices_all],
+        ["nmap_scan", sql_nmap_scan_all],
+        ["pholus_scan", sql_pholus_scan_all],
+        ["events_pending_alert", sql_events_pending_alert],
+        ["settings", sql_settings],
+        ["plugins_events", sql_plugins_events],
+        ["plugins_history", sql_plugins_history],
+        ["plugins_objects", sql_plugins_objects],
+        ["language_strings", sql_language_strings],
+        ["custom_endpoint", conf.API_CUSTOM_SQL],
+    ]
+
+    # Save selected database tables
+    for dsSQL in dataSourcesSQLs:
+
+        if updateOnlyDataSources == [] or dsSQL[0] in updateOnlyDataSources:
+
+            api_endpoint_class(db, dsSQL[1], folder + 'table_' + dsSQL[0] + '.json')
+
+
+#-------------------------------------------------------------------------------
+
+
+class api_endpoint_class:
+    def __init__(self, db, query, path):        
+
+        global apiEndpoints
+        self.db = db
+        self.query = query
+        self.jsonData = db.get_table_as_json(self.query).json
+        self.path = path
+        self.fileName = path.split('/')[-1]
+        self.hash = hash(json.dumps(self.jsonData))
+
+        # check if the endpoint needs to be updated
+        found = False
+        changed = False
+        changedIndex = -1
+        index = 0
+        
+        # search previous endpoint states to check if API needs updating
+        for endpoint in apiEndpoints:
+            # match sql and API endpoint path 
+            if endpoint.query == self.query and endpoint.path == self.path:
+                found = True 
+                if endpoint.hash != self.hash:                    
+                    changed = True
+                    changedIndex = index
+
+            index = index + 1
+        
+        # cehck if API endpoints have changed or if it's a new one
+        if not found or changed:
+
+            mylog('verbose', [f'[API] Updating {self.fileName} file in /front/api'])
+
+            write_file(self.path, json.dumps(self.jsonData)) 
+            
+            if not found:
+                apiEndpoints.append(self)
+
+            elif changed and changedIndex != -1 and changedIndex < len(apiEndpoints):
+                # update hash
+                apiEndpoints[changedIndex].hash = self.hash
+            else:
+                mylog('info', [f'[API] ERROR Updating {self.fileName}'])
+            

pialert/conf.py

@@ -0,0 +1,125 @@
+""" config related functions for Pi.Alert """
+
+
+# These are global variables, not config items and should not exist !
+mySettings = []
+mySettingsSQLsafe = []
+debug_force_notification = False
+cycle = 1
+userSubnets = []
+mySchedules = [] # bad solution for global - TO-DO
+plugins = []  # bad solution for global - TO-DO
+tz = ''
+
+# modified time of the most recently imported config file
+# set to a small value to force import at first run
+lastImportedConfFile = 1.1 
+
+plugins_once_run = False
+newVersionAvailable = False
+time_started = ''
+check_report = []
+log_timestamp = 0
+arpscan_devices = []
+# for MQTT
+mqtt_connected_to_broker = False
+mqtt_sensors = []
+client = None  # mqtt client
+# for notifications
+changedPorts_json_struc = None
+
+
+
+# ACTUAL CONFIGRATION ITEMS set to defaults
+
+# General
+ENABLE_ARPSCAN = True 
+SCAN_SUBNETS = ['192.168.1.0/24 --interface=eth1', '192.168.1.0/24 --interface=eth0']   
+LOG_LEVEL = 'verbose' 
+TIMEZONE = 'Europe/Berlin'
+ENABLE_PLUGINS =  True 
+PIALERT_WEB_PROTECTION =  False 
+PIALERT_WEB_PASSWORD = '8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92' 
+INCLUDED_SECTIONS =  ['internet', 'new_devices', 'down_devices', 'events', 'ports']   
+SCAN_CYCLE_MINUTES = 5 
+DAYS_TO_KEEP_EVENTS =  90 
+REPORT_DASHBOARD_URL =  'http://pi.alert/' 
+DIG_GET_IP_ARG = '-4 myip.opendns.com @resolver1.opendns.com' 
+UI_LANG = 'English' 
+UI_PRESENCE =  ['online', 'offline', 'archived']  
+
+
+
+# Email
+REPORT_MAIL =  False 
+SMTP_SERVER =  '' 
+SMTP_PORT = 587 
+REPORT_TO = 'user@gmail.com'
+REPORT_FROM =  'Pi.Alert <user@gmail.com>' 
+SMTP_SKIP_LOGIN =  False 
+SMTP_USER = '' 
+SMTP_PASS = '' 
+SMTP_SKIP_TLS =  False 
+SMTP_FORCE_SSL =  False 
+
+# Webhooks
+REPORT_WEBHOOK = False 
+WEBHOOK_URL = '' 
+WEBHOOK_PAYLOAD = 'json' 
+WEBHOOK_REQUEST_METHOD = 'GET' 
+
+# Apprise
+REPORT_APPRISE = False 
+APPRISE_HOST = '' 
+APPRISE_URL = '' 
+APPRISE_PAYLOAD = 'html' 
+
+# NTFY
+REPORT_NTFY =  False 
+NTFY_HOST ='https://ntfy.sh'
+NTFY_TOPIC ='' 
+NTFY_USER = '' 
+NTFY_PASSWORD = '' 
+
+# PUSHSAFER
+REPORT_PUSHSAFER = False 
+PUSHSAFER_TOKEN = 'ApiKey'
+
+# MQTT
+REPORT_MQTT = False 
+MQTT_BROKER = '' 
+MQTT_PORT =  1883 
+MQTT_USER =  '' 
+MQTT_PASSWORD = '' 
+MQTT_QOS =  0 
+MQTT_DELAY_SEC =  2 
+
+# DynDNS
+DDNS_ACTIVE = False 
+DDNS_DOMAIN = 'your_domain.freeddns.org'
+DDNS_USER = 'dynu_user' 
+DDNS_PASSWORD = 'A0000000B0000000C0000000D0000000' 
+DDNS_UPDATE_URL = 'https://api.dynu.com/nic/update?'
+
+# PiHole
+PIHOLE_ACTIVE = False
+DHCP_ACTIVE = False 
+
+# PHOLUS
+PHOLUS_ACTIVE = False
+PHOLUS_TIMEOUT = 20 
+PHOLUS_FORCE = False
+PHOLUS_RUN = 'once'
+PHOLUS_RUN_TIMEOUT = 600
+PHOLUS_RUN_SCHD = '0 4 * * *'
+PHOLUS_DAYS_DATA = 0 
+
+# Nmap
+NMAP_ACTIVE =  True 
+NMAP_TIMEOUT =  150 
+NMAP_RUN = 'once' 
+NMAP_RUN_SCHD =  '0 2 * * *'
+NMAP_ARGS = '-p -10000 --max-parallelism 100'
+
+# API     
+API_CUSTOM_SQL = 'SELECT * FROM Devices WHERE dev_PresentLastScan = 0'

pialert/const.py

@@ -0,0 +1,54 @@
+""" CONSTANTS for Pi.Alert """
+
+#===============================================================================
+# PATHS
+#===============================================================================
+pialertPath = '/home/pi/pialert'
+#pialertPath ='/home/roland/repos/Pi.Alert'
+
+confPath = "/config/pialert.conf"
+dbPath = '/db/pialert.db'
+
+
+pluginsPath  = pialertPath + '/front/plugins'
+logPath      = pialertPath + '/front/log'
+apiPath      = pialertPath + '/front/api/'
+fullConfPath = pialertPath + confPath
+fullDbPath   = pialertPath + dbPath
+fullPholusPath = pialertPath+'/pholus/pholus3.py'
+
+
+vendorsDB              = '/usr/share/arp-scan/ieee-oui.txt'
+piholeDB               = '/etc/pihole/pihole-FTL.db'
+piholeDhcpleases       = '/etc/pihole/dhcp.leases'
+
+
+#===============================================================================
+# SQL queries
+#===============================================================================
+sql_devices_all = """select dev_MAC, dev_Name, dev_DeviceType, dev_Vendor, dev_Group, 
+                     dev_FirstConnection, dev_LastConnection, dev_LastIP, dev_StaticIP, 
+                     dev_PresentLastScan, dev_LastNotification, dev_NewDevice, 
+                     dev_Network_Node_MAC_ADDR, dev_Network_Node_port,  
+                     dev_Icon from Devices"""
+sql_devices_stats =  """SELECT Online_Devices as online, Down_Devices as down, All_Devices as 'all', Archived_Devices as archived, 
+                        (select count(*) from Devices a where dev_NewDevice = 1 ) as new, 
+                        (select count(*) from Devices a where dev_Name = '(unknown)' or dev_Name = '(name not found)' ) as unknown 
+                        from Online_History order by Scan_Date desc limit 1"""
+sql_nmap_scan_all = "SELECT  * FROM Nmap_Scan"
+sql_pholus_scan_all = "SELECT  * FROM Pholus_Scan"
+sql_events_pending_alert = "SELECT  * FROM Events where eve_PendingAlertEmail is not 0"
+sql_settings = "SELECT  * FROM Settings"
+sql_plugins_objects = "SELECT  * FROM Plugins_Objects"
+sql_language_strings = "SELECT  * FROM Plugins_Language_Strings"
+sql_plugins_events = "SELECT  * FROM Plugins_Events"
+sql_plugins_history = "SELECT  * FROM Plugins_History ORDER BY 'Index' DESC"
+sql_new_devices = """SELECT * FROM ( 
+                        SELECT eve_IP as dev_LastIP, eve_MAC as dev_MAC 
+                        FROM Events_Devices
+                        WHERE eve_PendingAlertEmail = 1
+                        AND eve_EventType = 'New Device'
+                        ORDER BY eve_DateTime ) t1
+                        LEFT JOIN 
+                        ( SELECT dev_Name, dev_MAC as dev_MAC_t2 FROM Devices) t2 
+                        ON t1.dev_MAC = t2.dev_MAC_t2"""

pialert/database.py

@@ -0,0 +1,472 @@
+""" all things database to support Pi.Alert """
+
+import sqlite3
+
+# pialert modules
+from const import fullDbPath, sql_devices_stats, sql_devices_all
+
+from logger import mylog
+from helper import json_struc, initOrSetParam, row_to_json, timeNow #, updateState
+
+
+
+
+
+
+
+class DB():
+    """
+    DB Class to provide the basic database interactions.
+    Open / Commit / Close / read / write
+    """
+
+    def __init__(self):
+        self.sql = None
+        self.sql_connection = None
+
+    #-------------------------------------------------------------------------------
+    def open (self):
+        # Check if DB is open
+        if self.sql_connection != None :
+            mylog('debug','openDB: databse already open')
+            return
+
+        mylog('none', '[Database] Opening DB' )
+        # Open DB and Cursor
+        try:
+            self.sql_connection = sqlite3.connect (fullDbPath, isolation_level=None)
+            self.sql_connection.execute('pragma journal_mode=wal') #
+            self.sql_connection.text_factory = str
+            self.sql_connection.row_factory = sqlite3.Row
+            self.sql = self.sql_connection.cursor()
+        except sqlite3.Error as e:
+            mylog('none',[ '[Database] - Open DB Error: ', e])
+
+
+    #-------------------------------------------------------------------------------
+    def commitDB (self):
+        if self.sql_connection == None :
+            mylog('debug','commitDB: databse is not open')
+            return False
+
+        # Commit changes to DB
+        self.sql_connection.commit()
+        return True
+
+    #-------------------------------------------------------------------------------
+    def get_sql_array(self, query):
+        if self.sql_connection == None :
+            mylog('debug','getQueryArray: databse is not open')
+            return
+
+        self.sql.execute(query)
+        rows = self.sql.fetchall()
+        #self.commitDB()
+
+        #  convert result into list of lists
+        arr = []
+        for row in rows:
+            r_temp = []
+            for column in row:
+                r_temp.append(column)
+            arr.append(r_temp)
+
+        return arr
+
+    #===============================================================================
+    # Cleanup / upkeep database
+    #===============================================================================
+    def cleanup_database (self, startTime, DAYS_TO_KEEP_EVENTS, PHOLUS_DAYS_DATA):
+        """
+        Cleaning out old records from the tables that don't need to keep all data.
+        """
+        # Header
+        #updateState(self,"Upkeep: Clean DB")
+        mylog('verbose', ['[DB Cleanup] Upkeep Database:' ])
+
+        # Cleanup Online History
+        mylog('verbose', ['[DB Cleanup] Online_History: Delete all but keep latest 150 entries'])
+        self.sql.execute ("""DELETE from Online_History where "Index" not in (
+                             SELECT "Index" from Online_History 
+                             order by Scan_Date desc limit 150)""")
+        mylog('verbose', ['[DB Cleanup] Optimize Database'])
+        # Cleanup Events
+        mylog('verbose', ['[DB Cleanup] Events: Delete all older than '+str(DAYS_TO_KEEP_EVENTS)+' days'])
+        self.sql.execute ("""DELETE FROM Events 
+                             WHERE eve_DateTime <= date('now', '-"+str(DAYS_TO_KEEP_EVENTS)+" day')""")
+        # Cleanup Plugin Events History
+        mylog('verbose', ['[DB Cleanup] Plugin Events History: Delete all older than '+str(DAYS_TO_KEEP_EVENTS)+' days'])
+        self.sql.execute ("""DELETE FROM Plugins_History 
+                             WHERE DateTimeChanged <= date('now', '-"+str(DAYS_TO_KEEP_EVENTS)+" day')""")
+        # Cleanup Pholus_Scan
+        if PHOLUS_DAYS_DATA != 0:
+            mylog('verbose', ['[DB Cleanup] Pholus_Scan: Delete all older than ' + str(PHOLUS_DAYS_DATA) + ' days'])
+            # improvement possibility: keep at least N per mac
+            self.sql.execute ("""DELETE FROM Pholus_Scan 
+                                 WHERE Time <= date('now', '-"+ str(PHOLUS_DAYS_DATA) +" day')""") 
+
+        # De-Dupe (de-duplicate - remove duplicate entries) from the Pholus_Scan table
+        mylog('verbose', ['[DB Cleanup] Pholus_Scan: Delete all duplicates'])
+        self.sql.execute ("""DELETE  FROM Pholus_Scan
+                        WHERE rowid > (
+                        SELECT MIN(rowid) FROM Pholus_Scan p2
+                        WHERE Pholus_Scan.MAC = p2.MAC
+                        AND Pholus_Scan.Value = p2.Value
+                        AND Pholus_Scan.Record_Type = p2.Record_Type
+                        );""")
+        # De-Dupe (de-duplicate - remove duplicate entries) from the Nmap_Scan table
+        mylog('verbose', ['    Nmap_Scan: Delete all duplicates'])
+        self.sql.execute ("""DELETE  FROM Nmap_Scan
+                        WHERE rowid > (
+                        SELECT MIN(rowid) FROM Nmap_Scan p2
+                        WHERE Nmap_Scan.MAC = p2.MAC
+                        AND Nmap_Scan.Port = p2.Port
+                        AND Nmap_Scan.State = p2.State
+                        AND Nmap_Scan.Service = p2.Service
+                        );""")
+
+        # Shrink DB
+        mylog('verbose', ['    Shrink Database'])
+        self.sql.execute ("VACUUM;")
+        self.commitDB()
+
+    #-------------------------------------------------------------------------------
+    def upgradeDB(self):
+        """
+        Check the current tables in the DB and upgrade them if neccessary
+        """
+
+        # indicates, if Online_History table is available
+        onlineHistoryAvailable = self.sql.execute("""
+        SELECT name FROM sqlite_master WHERE type='table'
+        AND name='Online_History';
+        """).fetchall() != []
+
+        # Check if it is incompatible (Check if table has all required columns)
+        isIncompatible = False
+
+        if onlineHistoryAvailable :
+          isIncompatible = self.sql.execute ("""
+          SELECT COUNT(*) AS CNTREC FROM pragma_table_info('Online_History') WHERE name='Archived_Devices'
+          """).fetchone()[0] == 0
+
+        # Drop table if available, but incompatible
+        if onlineHistoryAvailable and isIncompatible:
+          mylog('none','[upgradeDB] Table is incompatible, Dropping the Online_History table')
+          self.sql.execute("DROP TABLE Online_History;")
+          onlineHistoryAvailable = False
+
+        if onlineHistoryAvailable == False :
+          self.sql.execute("""
+          CREATE TABLE "Online_History" (
+            "Index"	INTEGER,
+            "Scan_Date"	TEXT,
+            "Online_Devices"	INTEGER,
+            "Down_Devices"	INTEGER,
+            "All_Devices"	INTEGER,
+            "Archived_Devices" INTEGER,
+            PRIMARY KEY("Index" AUTOINCREMENT)
+          );
+          """)
+
+        # Alter Devices table
+        # dev_Network_Node_MAC_ADDR column
+        dev_Network_Node_MAC_ADDR_missing = self.sql.execute ("""
+          SELECT COUNT(*) AS CNTREC FROM pragma_table_info('Devices') WHERE name='dev_Network_Node_MAC_ADDR'
+          """).fetchone()[0] == 0
+
+        if dev_Network_Node_MAC_ADDR_missing :
+          mylog('verbose', ["[upgradeDB] Adding dev_Network_Node_MAC_ADDR to the Devices table"])
+          self.sql.execute("""
+          ALTER TABLE "Devices" ADD "dev_Network_Node_MAC_ADDR" TEXT
+          """)
+
+        # dev_Network_Node_port column
+        dev_Network_Node_port_missing = self.sql.execute ("""
+          SELECT COUNT(*) AS CNTREC FROM pragma_table_info('Devices') WHERE name='dev_Network_Node_port'
+          """).fetchone()[0] == 0
+
+        if dev_Network_Node_port_missing :
+          mylog('verbose', ["[upgradeDB] Adding dev_Network_Node_port to the Devices table"])
+          self.sql.execute("""
+          ALTER TABLE "Devices" ADD "dev_Network_Node_port" INTEGER
+          """)
+
+        # dev_Icon column
+        dev_Icon_missing = self.sql.execute ("""
+          SELECT COUNT(*) AS CNTREC FROM pragma_table_info('Devices') WHERE name='dev_Icon'
+          """).fetchone()[0] == 0
+
+        if dev_Icon_missing :
+          mylog('verbose', ["[upgradeDB] Adding dev_Icon to the Devices table"])
+          self.sql.execute("""
+          ALTER TABLE "Devices" ADD "dev_Icon" TEXT
+          """)
+
+        # indicates, if Settings table is available
+        settingsMissing = self.sql.execute("""
+        SELECT name FROM sqlite_master WHERE type='table'
+        AND name='Settings';
+        """).fetchone() == None
+
+        # Re-creating Settings table
+        mylog('verbose', ["[upgradeDB] Re-creating Settings table"])
+
+        if settingsMissing == False:
+            self.sql.execute("DROP TABLE Settings;")
+
+        self.sql.execute("""
+        CREATE TABLE "Settings" (
+        "Code_Name"	    TEXT,
+        "Display_Name"	TEXT,
+        "Description"	TEXT,
+        "Type"          TEXT,
+        "Options"       TEXT,
+        "RegEx"         TEXT,
+        "Value"	        TEXT,
+        "Group"	        TEXT,
+        "Events"	    TEXT
+        );
+        """)
+
+        # indicates, if Pholus_Scan table is available
+        pholusScanMissing = self.sql.execute("""
+        SELECT name FROM sqlite_master WHERE type='table'
+        AND name='Pholus_Scan';
+        """).fetchone() == None
+
+        # if pholusScanMissing == False:
+        #     # Re-creating Pholus_Scan table
+        #     self.sql.execute("DROP TABLE Pholus_Scan;")
+        #     pholusScanMissing = True
+
+        if pholusScanMissing:
+            mylog('verbose', ["[upgradeDB] Re-creating Pholus_Scan table"])
+            self.sql.execute("""
+            CREATE TABLE "Pholus_Scan" (
+            "Index"	          INTEGER,
+            "Info"	          TEXT,
+            "Time"	          TEXT,
+            "MAC"	          TEXT,
+            "IP_v4_or_v6"	  TEXT,
+            "Record_Type"	  TEXT,
+            "Value"           TEXT,
+            "Extra"           TEXT,
+            PRIMARY KEY("Index" AUTOINCREMENT)
+            );
+            """)
+
+        # indicates, if Nmap_Scan table is available
+        nmapScanMissing = self.sql.execute("""
+        SELECT name FROM sqlite_master WHERE type='table'
+        AND name='Nmap_Scan';
+        """).fetchone() == None
+
+        # Re-creating Parameters table
+        mylog('verbose', ["[upgradeDB] Re-creating Parameters table"])
+        self.sql.execute("DROP TABLE Parameters;")
+
+        self.sql.execute("""
+          CREATE TABLE "Parameters" (
+            "par_ID" TEXT PRIMARY KEY,
+            "par_Value"	TEXT
+          );
+          """)
+
+        # Initialize Parameters if unavailable
+        initOrSetParam(self, 'Back_App_State','Initializing')
+
+        # if nmapScanMissing == False:
+        #     # Re-creating Nmap_Scan table
+        #     self.sql.execute("DROP TABLE Nmap_Scan;")
+        #     nmapScanMissing = True
+
+        if nmapScanMissing:
+            mylog('verbose', ["[upgradeDB] Re-creating Nmap_Scan table"])
+            self.sql.execute("""
+            CREATE TABLE "Nmap_Scan" (
+            "Index"	          INTEGER,
+            "MAC"	          TEXT,
+            "Port"	          TEXT,
+            "Time"	          TEXT,
+            "State"	          TEXT,
+            "Service"	      TEXT,
+            "Extra"           TEXT,
+            PRIMARY KEY("Index" AUTOINCREMENT)
+            );
+            """)
+
+        # Plugin state
+        sql_Plugins_Objects = """ CREATE TABLE IF NOT EXISTS Plugins_Objects(
+                            "Index"	          INTEGER,
+                            Plugin TEXT NOT NULL,
+                            Object_PrimaryID TEXT NOT NULL,
+                            Object_SecondaryID TEXT NOT NULL,
+                            DateTimeCreated TEXT NOT NULL,
+                            DateTimeChanged TEXT NOT NULL,
+                            Watched_Value1 TEXT NOT NULL,
+                            Watched_Value2 TEXT NOT NULL,
+                            Watched_Value3 TEXT NOT NULL,
+                            Watched_Value4 TEXT NOT NULL,
+                            Status TEXT NOT NULL,
+                            Extra TEXT NOT NULL,
+                            UserData TEXT NOT NULL,
+                            ForeignKey TEXT NOT NULL,
+                            PRIMARY KEY("Index" AUTOINCREMENT)
+                        ); """
+        self.sql.execute(sql_Plugins_Objects)
+
+        # Plugin execution results
+        sql_Plugins_Events = """ CREATE TABLE IF NOT EXISTS Plugins_Events(
+                            "Index"	          INTEGER,
+                            Plugin TEXT NOT NULL,
+                            Object_PrimaryID TEXT NOT NULL,
+                            Object_SecondaryID TEXT NOT NULL,
+                            DateTimeCreated TEXT NOT NULL,
+                            DateTimeChanged TEXT NOT NULL,
+                            Watched_Value1 TEXT NOT NULL,
+                            Watched_Value2 TEXT NOT NULL,
+                            Watched_Value3 TEXT NOT NULL,
+                            Watched_Value4 TEXT NOT NULL,
+                            Status TEXT NOT NULL,
+                            Extra TEXT NOT NULL,
+                            UserData TEXT NOT NULL,
+                            ForeignKey TEXT NOT NULL,
+                            PRIMARY KEY("Index" AUTOINCREMENT)
+                        ); """
+        self.sql.execute(sql_Plugins_Events)
+
+        # Plugin execution history
+        sql_Plugins_History = """ CREATE TABLE IF NOT EXISTS Plugins_History(
+                            "Index"	          INTEGER,
+                            Plugin TEXT NOT NULL,
+                            Object_PrimaryID TEXT NOT NULL,
+                            Object_SecondaryID TEXT NOT NULL,
+                            DateTimeCreated TEXT NOT NULL,
+                            DateTimeChanged TEXT NOT NULL,
+                            Watched_Value1 TEXT NOT NULL,
+                            Watched_Value2 TEXT NOT NULL,
+                            Watched_Value3 TEXT NOT NULL,
+                            Watched_Value4 TEXT NOT NULL,
+                            Status TEXT NOT NULL,
+                            Extra TEXT NOT NULL,
+                            UserData TEXT NOT NULL,
+                            ForeignKey TEXT NOT NULL,
+                            PRIMARY KEY("Index" AUTOINCREMENT)
+                        ); """
+        self.sql.execute(sql_Plugins_History)
+
+        # Dynamically generated language strings
+        # indicates, if Language_Strings table is available
+        languageStringsMissing = self.sql.execute("""
+        SELECT name FROM sqlite_master WHERE type='table'
+        AND name='Plugins_Language_Strings';
+        """).fetchone() == None
+
+        if languageStringsMissing == False:
+            self.sql.execute("DROP TABLE Plugins_Language_Strings;")
+
+        self.sql.execute(""" CREATE TABLE IF NOT EXISTS Plugins_Language_Strings(
+                            "Index"	          INTEGER,
+                            Language_Code TEXT NOT NULL,
+                            String_Key TEXT NOT NULL,
+                            String_Value TEXT NOT NULL,
+                            Extra TEXT NOT NULL,
+                            PRIMARY KEY("Index" AUTOINCREMENT)
+                        ); """)
+
+        self.commitDB()
+
+
+    #-------------------------------------------------------------------------------
+    def get_table_as_json(self, sqlQuery):
+
+        mylog('debug',[ '[Database] - get_table_as_json - Query: ', sqlQuery])
+        try:
+            self.sql.execute(sqlQuery)
+            columnNames = list(map(lambda x: x[0], self.sql.description))
+            rows = self.sql.fetchall()
+        except sqlite3.Error as e:
+            mylog('none',[ '[Database] - SQL ERROR: ', e])
+            return None
+
+        result = {"data":[]}
+        for row in rows:
+            tmp = row_to_json(columnNames, row)
+            result["data"].append(tmp)
+
+        mylog('debug',[ '[Database] - get_table_as_json - returning ', len(rows), " rows with columns: ", columnNames])
+        return json_struc(result, columnNames)
+
+    #-------------------------------------------------------------------------------
+    # referece from here: https://codereview.stackexchange.com/questions/241043/interface-class-for-sqlite-databases
+    #-------------------------------------------------------------------------------
+    def read(self, query, *args):
+        """check the query and arguments are aligned and are read only"""
+        mylog('debug',[ '[Database] - Read All: SELECT Query: ', query, " params: ", args])
+        try:
+            assert query.count('?') == len(args)
+            assert query.upper().strip().startswith('SELECT')
+            self.sql.execute(query, args)
+            rows = self.sql.fetchall()
+            return rows
+        except AssertionError:
+            mylog('none',[ '[Database] - ERROR: inconsistent query and/or arguments.', query, " params: ", args])
+        except sqlite3.Error as e:
+            mylog('none',[ '[Database] - SQL ERROR: ', e])
+        return None
+
+    def read_one(self, query, *args):
+        """ 
+        call read() with the same arguments but only returns the first row.
+        should only be used when there is a single row result expected
+        """
+
+        mylog('debug',[ '[Database] - Read One: ', query, " params: ", args])
+        rows = self.read(query, *args)
+        if len(rows) == 1:
+            return rows[0]
+                
+        if len(rows) > 1: 
+            mylog('none',[ '[Database] - Warning!: query returns multiple rows, only first row is passed on!', query, " params: ", args])
+            return rows[0]
+        # empty result set
+        return None
+
+
+
+#-------------------------------------------------------------------------------
+def get_device_stats(db):
+    # columns = ["online","down","all","archived","new","unknown"]
+    return db.read_one(sql_devices_stats)
+#-------------------------------------------------------------------------------
+def get_all_devices(db):
+    return db.read(sql_devices_all)
+
+#-------------------------------------------------------------------------------
+
+#-------------------------------------------------------------------------------
+def insertOnlineHistory(db):
+    sql = db.sql #TO-DO
+    startTime = timeNow()
+    # Add to History
+
+    # only run this if the scans have run
+    scanCount = db.read_one("SELECT count(*) FROM CurrentScan")
+    if scanCount[0] == 0 :
+        mylog('debug',[ '[insertOnlineHistory] - nothing to do, currentScan empty'])
+        return 0
+
+    History_All = db.read("SELECT * FROM Devices")
+    History_All_Devices  = len(History_All)
+
+    History_Archived = db.read("SELECT * FROM Devices WHERE dev_Archived = 1")
+    History_Archived_Devices  = len(History_Archived)
+
+    History_Online = db.read("SELECT * FROM CurrentScan")
+    History_Online_Devices  = len(History_Online)
+    History_Offline_Devices = History_All_Devices - History_Archived_Devices - History_Online_Devices
+
+    sql.execute ("INSERT INTO Online_History (Scan_Date, Online_Devices, Down_Devices, All_Devices, Archived_Devices) "+
+                 "VALUES ( ?, ?, ?, ?, ?)", (startTime, History_Online_Devices, History_Offline_Devices, History_All_Devices, History_Archived_Devices ) )
+    db.commitDB()

pialert/device.py

@@ -0,0 +1,432 @@
+
+
+
+
+import subprocess
+
+import conf
+from helper import timeNow
+from scanners.internet import check_IP_format, get_internet_IP
+from logger import mylog, print_log
+from mac_vendor import query_MAC_vendor
+from scanners.pholusscan import performPholusScan, resolve_device_name_dig, resolve_device_name_pholus
+#-------------------------------------------------------------------------------
+
+
+def save_scanned_devices (db, p_arpscan_devices, p_cycle_interval):
+    sql = db.sql #TO-DO
+    cycle = 1 # always 1, only one cycle supported
+
+    # Delete previous scan data
+    sql.execute ("DELETE FROM CurrentScan WHERE cur_ScanCycle = ?",
+                (cycle,))
+
+    if len(p_arpscan_devices) > 0:
+        # Insert new arp-scan devices
+        sql.executemany ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, "+
+                        "    cur_IP, cur_Vendor, cur_ScanMethod) "+
+                        "VALUES ("+ str(cycle) + ", :mac, :ip, :hw, 'arp-scan')",
+                        p_arpscan_devices) 
+
+    # Insert Pi-hole devices
+    startTime = timeNow()
+    sql.execute ("""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, 
+                        cur_IP, cur_Vendor, cur_ScanMethod)
+                    SELECT ?, PH_MAC, PH_IP, PH_Vendor, 'Pi-hole'
+                    FROM PiHole_Network
+                    WHERE PH_LastQuery >= ?
+                      AND NOT EXISTS (SELECT 'X' FROM CurrentScan
+                                      WHERE cur_MAC = PH_MAC
+                                        AND cur_ScanCycle = ? )""",
+                    (cycle,
+                     (int(startTime.strftime('%s')) - 60 * p_cycle_interval),
+                     cycle) )
+
+    # Check Internet connectivity
+    internet_IP = get_internet_IP( conf.DIG_GET_IP_ARG )
+        # TESTING - Force IP
+        # internet_IP = ""
+    if internet_IP != "" :
+        sql.execute ("""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod)
+                        VALUES (?, 'Internet', ?, Null, 'queryDNS') """, (cycle, internet_IP) )
+
+    # #76 Add Local MAC of default local interface
+      # BUGFIX #106 - Device that pialert is running
+        # local_mac_cmd = ["bash -lc ifconfig `ip route list default | awk {'print $5'}` | grep ether | awk '{print $2}'"]
+          # local_mac_cmd = ["/sbin/ifconfig `ip route list default | sort -nk11 | head -1 | awk {'print $5'}` | grep ether | awk '{print $2}'"]
+    local_mac_cmd = ["/sbin/ifconfig `ip -o route get 1 | sed 's/^.*dev \\([^ ]*\\).*$/\\1/;q'` | grep ether | awk '{print $2}'"]
+    local_mac = subprocess.Popen (local_mac_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()[0].decode().strip()
+
+    # local_dev_cmd = ["ip -o route get 1 | sed 's/^.*dev \\([^ ]*\\).*$/\\1/;q'"]
+    # local_dev = subprocess.Popen (local_dev_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()[0].decode().strip()
+    
+    # local_ip_cmd = ["ip route list default | awk {'print $7'}"]
+    local_ip_cmd = ["ip -o route get 1 | sed 's/^.*src \\([^ ]*\\).*$/\\1/;q'"]
+    local_ip = subprocess.Popen (local_ip_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()[0].decode().strip()
+
+    mylog('debug', ['[Save Devices] Saving this IP into the CurrentScan table:', local_ip])
+
+    if check_IP_format(local_ip) == '':
+        local_ip = '0.0.0.0'
+
+    # Check if local mac has been detected with other methods
+    sql.execute ("SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanCycle = ? AND cur_MAC = ? ", (cycle, local_mac) )
+    if sql.fetchone()[0] == 0 :
+        sql.execute ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod) "+
+                     "VALUES ( ?, ?, ?, Null, 'local_MAC') ", (cycle, local_mac, local_ip) )
+
+#-------------------------------------------------------------------------------
+def print_scan_stats (db):
+    sql = db.sql #TO-DO
+    # Devices Detected
+    sql.execute ("""SELECT COUNT(*) FROM CurrentScan
+                    WHERE cur_ScanCycle = ? """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]    Devices Detected.......: ', str (sql.fetchone()[0]) ])
+
+    # Devices arp-scan
+    sql.execute ("""SELECT COUNT(*) FROM CurrentScan
+                    WHERE cur_ScanMethod='arp-scan' AND cur_ScanCycle = ? """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        arp-scan detected..: ', str (sql.fetchone()[0]) ])
+
+    # Devices Pi-hole
+    sql.execute ("""SELECT COUNT(*) FROM CurrentScan
+                    WHERE cur_ScanMethod='PiHole' AND cur_ScanCycle = ? """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        Pi-hole detected...: +' + str (sql.fetchone()[0]) ])
+
+    # New Devices
+    sql.execute ("""SELECT COUNT(*) FROM CurrentScan
+                    WHERE cur_ScanCycle = ? 
+                      AND NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = cur_MAC) """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        New Devices........: ' + str (sql.fetchone()[0]) ])
+
+    # Devices in this ScanCycle
+    sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
+                    WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
+                      AND dev_ScanCycle = ? """,
+                    (conf.cycle,))
+    
+    mylog('verbose', ['[Scan Stats]    Devices in this cycle..: ' + str (sql.fetchone()[0]) ])
+
+    # Down Alerts
+    sql.execute ("""SELECT COUNT(*) FROM Devices
+                    WHERE dev_AlertDeviceDown = 1
+                      AND dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        Down Alerts........: ' + str (sql.fetchone()[0]) ])
+
+    # New Down Alerts
+    sql.execute ("""SELECT COUNT(*) FROM Devices
+                    WHERE dev_AlertDeviceDown = 1
+                      AND dev_PresentLastScan = 1
+                      AND dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        New Down Alerts....: ' + str (sql.fetchone()[0]) ])
+
+    # New Connections
+    sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
+                    WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
+                      AND dev_PresentLastScan = 0
+                      AND dev_ScanCycle = ? """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        New Connections....: ' + str ( sql.fetchone()[0]) ])
+
+    # Disconnections
+    sql.execute ("""SELECT COUNT(*) FROM Devices
+                    WHERE dev_PresentLastScan = 1
+                      AND dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        Disconnections.....: ' + str ( sql.fetchone()[0]) ])
+
+    # IP Changes
+    sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
+                    WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
+                      AND dev_ScanCycle = ?
+                      AND dev_LastIP <> cur_IP """,
+                    (conf.cycle,))
+    mylog('verbose', ['[Scan Stats]        IP Changes.........: ' + str ( sql.fetchone()[0]) ])
+
+
+
+#-------------------------------------------------------------------------------
+def create_new_devices (db):
+    sql = db.sql # TO-DO
+    startTime = timeNow()
+
+    # arpscan - Insert events for new devices
+    mylog('debug','[New Devices] New devices - 1 Events')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT cur_MAC, cur_IP, ?, 'New Device', cur_Vendor, 1
+                    FROM CurrentScan
+                    WHERE cur_ScanCycle = ? 
+                      AND NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = cur_MAC) """,
+                    (startTime, conf.cycle) ) 
+
+    mylog('debug','[New Devices] Insert Connection into session table')
+    sql.execute ("""INSERT INTO Sessions (ses_MAC, ses_IP, ses_EventTypeConnection, ses_DateTimeConnection,
+                        ses_EventTypeDisconnection, ses_DateTimeDisconnection, ses_StillConnected, ses_AdditionalInfo)
+                    SELECT cur_MAC, cur_IP,'Connected',?, NULL , NULL ,1, cur_Vendor
+                    FROM CurrentScan 
+                    WHERE cur_ScanCycle = ? 
+                      AND NOT EXISTS (SELECT 1 FROM Sessions
+                                      WHERE ses_MAC = cur_MAC) """,
+                    (startTime, conf.cycle) ) 
+                    
+    # arpscan - Create new devices
+    mylog('debug','[New Devices] 2 Create devices')
+    sql.execute ("""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
+                        dev_LastIP, dev_FirstConnection, dev_LastConnection,
+                        dev_ScanCycle, dev_AlertEvents, dev_AlertDeviceDown,
+                        dev_PresentLastScan)
+                    SELECT cur_MAC, '(unknown)', cur_Vendor, cur_IP, ?, ?,
+                        1, 1, 0, 1
+                    FROM CurrentScan
+                    WHERE cur_ScanCycle = ? 
+                      AND NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = cur_MAC) """,
+                    (startTime, startTime, conf.cycle) ) 
+
+    # Pi-hole - Insert events for new devices
+    # NOT STRICYLY NECESARY (Devices can be created through Current_Scan)
+    # Bugfix #2 - Pi-hole devices w/o IP
+    mylog('debug','[New Devices] 3 Pi-hole Events')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT PH_MAC, IFNULL (PH_IP,'-'), ?, 'New Device',
+                        '(Pi-Hole) ' || PH_Vendor, 1
+                    FROM PiHole_Network
+                    WHERE NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = PH_MAC) """,
+                    (startTime, ) ) 
+
+    # Pi-hole - Create New Devices
+    # Bugfix #2 - Pi-hole devices w/o IP
+    mylog('debug','[New Devices] 4 Pi-hole Create devices')
+    sql.execute ("""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
+                        dev_LastIP, dev_FirstConnection, dev_LastConnection,
+                        dev_ScanCycle, dev_AlertEvents, dev_AlertDeviceDown,
+                        dev_PresentLastScan)
+                    SELECT PH_MAC, PH_Name, PH_Vendor, IFNULL (PH_IP,'-'),
+                        ?, ?, 1, 1, 0, 1
+                    FROM PiHole_Network
+                    WHERE NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = PH_MAC) """,
+                    (startTime, startTime) ) 
+
+    # DHCP Leases - Insert events for new devices
+    mylog('debug','[New Devices] 5 DHCP Leases Events')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT DHCP_MAC, DHCP_IP, ?, 'New Device', '(DHCP lease)',1
+                    FROM DHCP_Leases
+                    WHERE NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = DHCP_MAC) """,
+                    (startTime, ) ) 
+
+    # DHCP Leases - Create New Devices
+    mylog('debug','[New Devices] 6 DHCP Leases Create devices')
+    # BUGFIX #23 - Duplicated MAC in DHCP.Leases
+    # TEST - Force Duplicated MAC
+        # sql.execute ("""INSERT INTO DHCP_Leases VALUES
+        #                 (1610700000, 'TEST1', '10.10.10.1', 'Test 1', '*')""")
+        # sql.execute ("""INSERT INTO DHCP_Leases VALUES
+        #                 (1610700000, 'TEST2', '10.10.10.2', 'Test 2', '*')""")
+    sql.execute ("""INSERT INTO Devices (dev_MAC, dev_name, dev_LastIP, 
+                        dev_Vendor, dev_FirstConnection, dev_LastConnection,
+                        dev_ScanCycle, dev_AlertEvents, dev_AlertDeviceDown,
+                        dev_PresentLastScan)
+                    SELECT DISTINCT DHCP_MAC,
+                        (SELECT DHCP_Name FROM DHCP_Leases AS D2
+                         WHERE D2.DHCP_MAC = D1.DHCP_MAC
+                         ORDER BY DHCP_DateTime DESC LIMIT 1),
+                        (SELECT DHCP_IP FROM DHCP_Leases AS D2
+                         WHERE D2.DHCP_MAC = D1.DHCP_MAC
+                         ORDER BY DHCP_DateTime DESC LIMIT 1),
+                        '(unknown)', ?, ?, 1, 1, 0, 1
+                    FROM DHCP_Leases AS D1
+                    WHERE NOT EXISTS (SELECT 1 FROM Devices
+                                      WHERE dev_MAC = DHCP_MAC) """,
+                    (startTime, startTime) ) 
+
+    # sql.execute ("""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
+    #                     dev_LastIP, dev_FirstConnection, dev_LastConnection,
+    #                     dev_ScanCycle, dev_AlertEvents, dev_AlertDeviceDown,
+    #                     dev_PresentLastScan)
+    #                 SELECT DHCP_MAC, DHCP_Name, '(unknown)', DHCP_IP, ?, ?,
+    #                     1, 1, 0, 1
+    #                 FROM DHCP_Leases
+    #                 WHERE NOT EXISTS (SELECT 1 FROM Devices
+    #                                   WHERE dev_MAC = DHCP_MAC) """,
+    #                 (startTime, startTime) ) 
+    mylog('debug','[New Devices] New Devices end')
+    db.commitDB()
+
+
+#-------------------------------------------------------------------------------
+def update_devices_data_from_scan (db):
+    sql = db.sql #TO-DO    
+    startTime = timeNow()
+    # Update Last Connection
+    mylog('debug','[Update Devices] 1 Last Connection')
+    sql.execute ("""UPDATE Devices SET dev_LastConnection = ?,
+                        dev_PresentLastScan = 1
+                    WHERE dev_ScanCycle = ?
+                      AND dev_PresentLastScan = 0
+                      AND EXISTS (SELECT 1 FROM CurrentScan 
+                                  WHERE dev_MAC = cur_MAC
+                                    AND dev_ScanCycle = cur_ScanCycle) """,
+                    (startTime, conf.cycle))
+
+    # Clean no active devices
+    mylog('debug','[Update Devices] 2 Clean no active devices')
+    sql.execute ("""UPDATE Devices SET dev_PresentLastScan = 0
+                    WHERE dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan 
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (conf.cycle,))
+
+    # Update IP & Vendor
+    mylog('debug','[Update Devices] - 3 LastIP & Vendor')
+    sql.execute ("""UPDATE Devices
+                    SET dev_LastIP = (SELECT cur_IP FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle),
+                        dev_Vendor = (SELECT cur_Vendor FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle)
+                    WHERE dev_ScanCycle = ?
+                      AND EXISTS (SELECT 1 FROM CurrentScan
+                                  WHERE dev_MAC = cur_MAC
+                                    AND dev_ScanCycle = cur_ScanCycle) """,
+                    (conf.cycle,)) 
+
+    # Pi-hole Network - Update (unknown) Name
+    mylog('debug','[Update Devices] - 4 Unknown Name')
+    sql.execute ("""UPDATE Devices
+                    SET dev_NAME = (SELECT PH_Name FROM PiHole_Network
+                                    WHERE PH_MAC = dev_MAC)
+                    WHERE (dev_Name in ("(unknown)", "(name not found)", "" )
+                           OR dev_Name IS NULL)
+                      AND EXISTS (SELECT 1 FROM PiHole_Network
+                                  WHERE PH_MAC = dev_MAC
+                                    AND PH_NAME IS NOT NULL
+                                    AND PH_NAME <> '') """)
+
+    # DHCP Leases - Update (unknown) Name
+    sql.execute ("""UPDATE Devices
+                    SET dev_NAME = (SELECT DHCP_Name FROM DHCP_Leases
+                                    WHERE DHCP_MAC = dev_MAC)
+                    WHERE (dev_Name in ("(unknown)", "(name not found)", "" )                           
+                           OR dev_Name IS NULL)
+                      AND EXISTS (SELECT 1 FROM DHCP_Leases
+                                  WHERE DHCP_MAC = dev_MAC)""")
+
+    # DHCP Leases - Vendor
+    mylog('debug','[Update Devices] - 5 Vendor')
+
+    recordsToUpdate = []
+    query = """SELECT * FROM Devices
+               WHERE dev_Vendor = '(unknown)' OR dev_Vendor =''
+                  OR dev_Vendor IS NULL"""
+
+    for device in sql.execute (query) :
+        vendor = query_MAC_vendor (device['dev_MAC'])
+        if vendor != -1 and vendor != -2 :
+            recordsToUpdate.append ([vendor, device['dev_MAC']])
+
+    sql.executemany ("UPDATE Devices SET dev_Vendor = ? WHERE dev_MAC = ? ",
+        recordsToUpdate )
+
+    # clean-up device leases table
+    sql.execute ("DELETE FROM DHCP_Leases")
+    mylog('debug','[Update Devices] Update devices end')
+
+#-------------------------------------------------------------------------------
+def update_devices_names (db):
+    sql = db.sql #TO-DO
+    # Initialize variables
+    recordsToUpdate = []
+    recordsNotFound = []
+
+    ignored = 0
+    notFound = 0
+
+    foundDig = 0
+    foundPholus = 0
+
+    # BUGFIX #97 - Updating name of Devices w/o IP    
+    sql.execute ("SELECT * FROM Devices WHERE dev_Name IN ('(unknown)','', '(name not found)') AND dev_LastIP <> '-'")
+    unknownDevices = sql.fetchall() 
+    db.commitDB()
+
+    # perform Pholus scan if (unknown) devices found
+    if conf.PHOLUS_ACTIVE and (len(unknownDevices) > 0 or conf.PHOLUS_FORCE):        
+        performPholusScan(db, conf.PHOLUS_TIMEOUT, conf.userSubnets)
+
+    # skip checks if no unknown devices
+    if len(unknownDevices) == 0 and conf.PHOLUS_FORCE == False:
+        return
+
+    # Devices without name
+    mylog('verbose', '[Update Device Name] Trying to resolve devices without name')
+
+    # get names from Pholus scan 
+    sql.execute ('SELECT * FROM Pholus_Scan where "Record_Type"="Answer"')    
+    pholusResults = list(sql.fetchall())        
+    db.commitDB()
+
+    # Number of entries from previous Pholus scans
+    mylog('verbose', ['[Update Device Name] Pholus entries from prev scans: ', len(pholusResults)])
+
+    for device in unknownDevices:
+        newName = -1
+        
+        # Resolve device name with DiG
+        newName = resolve_device_name_dig (device['dev_MAC'], device['dev_LastIP'])
+        
+        # count
+        if newName != -1:
+            foundDig += 1
+
+        # Resolve with Pholus 
+        if newName == -1:
+            newName =  resolve_device_name_pholus (device['dev_MAC'], device['dev_LastIP'], pholusResults)
+            # count
+            if newName != -1:
+                foundPholus += 1
+        
+        # isf still not found update name so we can distinguish the devices where we tried already
+        if newName == -1 :
+            recordsNotFound.append (["(name not found)", device['dev_MAC']])          
+        else:
+            # name wa sfound with DiG or Pholus
+            recordsToUpdate.append ([newName, device['dev_MAC']])
+
+    # Print log            
+    mylog('verbose', ['[Update Device Name] Names Found (DiG/Pholus): ', len(recordsToUpdate), " (",foundDig,"/",foundPholus ,")"] )                 
+    mylog('verbose', ['[Update Device Name] Names Not Found         : ', len(recordsNotFound)] )    
+     
+    # update not found devices with (name not found) 
+    sql.executemany ("UPDATE Devices SET dev_Name = ? WHERE dev_MAC = ? ", recordsNotFound )
+    # update names of devices which we were bale to resolve
+    sql.executemany ("UPDATE Devices SET dev_Name = ? WHERE dev_MAC = ? ", recordsToUpdate )
+    db.commitDB()

pialert/helper.py

@@ -0,0 +1,323 @@
+""" Colection of generic functions to support Pi.Alert """
+
+import io
+import sys
+import datetime
+import os
+import re
+import subprocess
+from pytz import timezone
+from datetime import timedelta
+import json
+import time
+from pathlib import Path
+import requests
+
+import conf
+from const import *
+from logger import mylog, logResult
+
+
+
+#-------------------------------------------------------------------------------
+def timeNow():
+    return datetime.datetime.now().replace(microsecond=0)
+#-------------------------------------------------------------------------------
+def timeNowTZ():
+    return datetime.datetime.now(conf.tz).replace(microsecond=0)
+
+#-------------------------------------------------------------------------------
+def updateState(db, newState):
+
+    # ?? Why is the state written to the DB?
+
+    #sql = db.sql
+
+    mylog('debug', '[updateState] changing state to: "' + newState +'"')
+    db.sql.execute ("UPDATE Parameters SET par_Value='"+ newState +"' WHERE par_ID='Back_App_State'")
+
+    db.commitDB()
+#-------------------------------------------------------------------------------
+def updateSubnets(scan_subnets):
+    subnets = []
+
+    # multiple interfaces
+    if type(scan_subnets) is list:
+        for interface in scan_subnets :
+            subnets.append(interface)
+    # one interface only
+    else:
+        subnets.append(scan_subnets)
+
+    return subnets
+
+
+
+#-------------------------------------------------------------------------------
+# check RW access of DB and config file
+def checkPermissionsOK():
+    #global confR_access, confW_access, dbR_access, dbW_access
+
+    confR_access = (os.access(fullConfPath, os.R_OK))
+    confW_access = (os.access(fullConfPath, os.W_OK))
+    dbR_access = (os.access(fullDbPath, os.R_OK))
+    dbW_access = (os.access(fullDbPath, os.W_OK))
+
+    mylog('none', ['\n'])
+    mylog('none', ['Permissions check (All should be True)'])
+    mylog('none', ['------------------------------------------------'])
+    mylog('none', [ "  " , confPath ,     " | " , " READ  | " , confR_access])
+    mylog('none', [ "  " , confPath ,     " | " , " WRITE | " , confW_access])
+    mylog('none', [ "  " , dbPath , "       | " , " READ  | " , dbR_access])
+    mylog('none', [ "  " , dbPath , "       | " , " WRITE | " , dbW_access])
+    mylog('none', ['------------------------------------------------'])
+
+    #return dbR_access and dbW_access and confR_access and confW_access
+    return (confR_access, dbR_access)
+#-------------------------------------------------------------------------------
+def fixPermissions():
+    # Try fixing access rights if needed
+    chmodCommands = []
+
+    chmodCommands.append(['sudo', 'chmod', 'a+rw', '-R', fullDbPath])
+    chmodCommands.append(['sudo', 'chmod', 'a+rw', '-R', fullConfPath])
+
+    for com in chmodCommands:
+        # Execute command
+        mylog('none', ["[Setup] Attempting to fix permissions."])
+        try:
+            # try runnning a subprocess
+            result = subprocess.check_output (com, universal_newlines=True)
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', ["[Setup] Fix Failed. Execute this command manually inside of the container: ", ' '.join(com)])
+            mylog('none', [e.output])
+
+
+#-------------------------------------------------------------------------------
+def initialiseFile(pathToCheck, defaultFile):
+    # if file not readable (missing?) try to copy over the backed-up (default) one
+    if str(os.access(pathToCheck, os.R_OK)) == "False":
+        mylog('none', ["[Setup] ("+ pathToCheck +") file is not readable or missing. Trying to copy over the default one."])
+        try:
+            # try runnning a subprocess
+            p = subprocess.Popen(["cp", defaultFile , pathToCheck], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            stdout, stderr = p.communicate()
+
+            if str(os.access(pathToCheck, os.R_OK)) == "False":
+                mylog('none', ["[Setup] Error copying ("+defaultFile+") to ("+pathToCheck+"). Make sure the app has Read & Write access to the parent directory."])
+            else:
+                mylog('none', ["[Setup] ("+defaultFile+") copied over successfully to ("+pathToCheck+")."])
+
+            # write stdout and stderr into .log files for debugging if needed
+            logResult (stdout, stderr)  # TO-DO should be changed to mylog
+
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', ["[Setup] Error copying ("+defaultFile+"). Make sure the app has Read & Write access to " + pathToCheck])
+            mylog('none', [e.output])
+
+
+def filePermissions():
+    # check and initialize pialert.conf
+    (confR_access, dbR_access) = checkPermissionsOK() # Initial check
+
+    if confR_access == False:
+        initialiseFile(fullConfPath, "/home/pi/pialert/back/pialert.conf_bak" )
+
+    # check and initialize pialert.db
+    if dbR_access == False:
+        initialiseFile(fullDbPath, "/home/pi/pialert/back/pialert.db_bak")
+
+    # last attempt
+    fixPermissions()
+
+
+#-------------------------------------------------------------------------------
+
+def bytes_to_string(value):
+    # if value is of type bytes, convert to string
+    if isinstance(value, bytes):
+        value = value.decode('utf-8')
+    return value
+
+#-------------------------------------------------------------------------------
+
+def if_byte_then_to_str(input):
+    if isinstance(input, bytes):
+        input = input.decode('utf-8')
+        input = bytes_to_string(re.sub('[^a-zA-Z0-9-_\s]', '', str(input)))
+    return input
+
+#-------------------------------------------------------------------------------
+def collect_lang_strings(db, json, pref):
+
+    for prop in json["localized"]:
+        for language_string in json[prop]:
+            import_language_string(db, language_string["language_code"], pref + "_" + prop, language_string["string"])
+
+
+#-------------------------------------------------------------------------------
+#  Creates a JSON object from a DB row
+def row_to_json(names, row):
+
+    rowEntry = {}
+
+    index = 0
+    for name in names:
+        rowEntry[name]= if_byte_then_to_str(row[name])
+        index += 1
+
+    return rowEntry
+
+#-------------------------------------------------------------------------------
+def import_language_string(db, code, key, value, extra = ""):
+
+    db.sql.execute ("""INSERT INTO Plugins_Language_Strings ("Language_Code", "String_Key", "String_Value", "Extra") VALUES (?, ?, ?, ?)""", (str(code), str(key), str(value), str(extra)))
+
+    db.commitDB()
+
+
+
+#-------------------------------------------------------------------------------
+def checkIPV4(ip):
+    """ Define a function to validate an Ip address
+    """
+    ipRegex = "^((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$"
+
+    if(re.search(ipRegex, ip)):
+        return True
+    else:
+        return False
+
+
+#-------------------------------------------------------------------------------
+def isNewVersion(newVersion: bool):
+
+    if newVersion == False:
+
+        f = open(pialertPath + '/front/buildtimestamp.txt', 'r')
+        buildTimestamp = int(f.read().strip())
+        f.close()
+
+        data = ""
+
+        try:
+            url = requests.get("https://api.github.com/repos/jokob-sk/Pi.Alert/releases")
+            text = url.text
+            data = json.loads(text)
+        except requests.exceptions.ConnectionError as e:
+            mylog('info', ["    Couldn't check for new release."])
+            data = ""
+
+        # make sure we received a valid response and not an API rate limit exceeded message
+        if data != "" and len(data) > 0 and isinstance(data, list) and "published_at" in data[0]:
+
+            dateTimeStr = data[0]["published_at"]
+
+            realeaseTimestamp = int(datetime.datetime.strptime(dateTimeStr, '%Y-%m-%dT%H:%M:%SZ').strftime('%s'))
+
+            if realeaseTimestamp > buildTimestamp + 600:
+                mylog('none', ["    New version of the container available!"])
+                newVersion = True
+                # updateState(db, 'Back_New_Version_Available', str(newVersionAvailable))     ## TO DO add this back in but avoid circular ref with database
+
+    return newVersion
+
+#-------------------------------------------------------------------------------
+def hide_email(email):
+    m = email.split('@')
+
+    if len(m) == 2:
+        return f'{m[0][0]}{"*"*(len(m[0])-2)}{m[0][-1] if len(m[0]) > 1 else ""}@{m[1]}'
+
+    return email
+
+#-------------------------------------------------------------------------------
+def removeDuplicateNewLines(text):
+    if "\n\n\n" in text:
+        return removeDuplicateNewLines(text.replace("\n\n\n", "\n\n"))
+    else:
+        return text
+
+#-------------------------------------------------------------------------------
+
+def add_json_list (row, list):
+    new_row = []
+    for column in row :
+        column = bytes_to_string(column)
+
+        new_row.append(column)
+
+    list.append(new_row)
+
+    return list
+
+#-------------------------------------------------------------------------------
+
+def sanitize_string(input):
+    if isinstance(input, bytes):
+        input = input.decode('utf-8')
+    value = bytes_to_string(re.sub('[^a-zA-Z0-9-_\s]', '', str(input)))
+    return value
+
+
+#-------------------------------------------------------------------------------
+def generate_mac_links (html, deviceUrl):
+
+    p = re.compile(r'(?:[0-9a-fA-F]:?){12}')
+
+    MACs = re.findall(p, html)
+
+    for mac in MACs:
+        html = html.replace('<td>' + mac + '</td>','<td><a href="' + deviceUrl + mac + '">' + mac + '</a></td>')
+
+    return html
+
+
+
+#-------------------------------------------------------------------------------
+def initOrSetParam(db, parID, parValue):
+    sql = db.sql
+
+    sql.execute ("INSERT INTO Parameters(par_ID, par_Value) VALUES('"+str(parID)+"', '"+str(parValue)+"') ON CONFLICT(par_ID) DO UPDATE SET par_Value='"+str(parValue)+"' where par_ID='"+str(parID)+"'")
+
+    db.commitDB()
+
+#-------------------------------------------------------------------------------
+class json_struc:
+    def __init__(self, jsn, columnNames):
+        self.json = jsn
+        self.columnNames = columnNames
+
+
+
+#-------------------------------------------------------------------------------
+def get_file_content(path):
+
+    f = open(path, 'r')
+    content = f.read()
+    f.close()
+
+    return content
+
+#-------------------------------------------------------------------------------
+def write_file (pPath, pText):
+    # Write the text depending using the correct python version
+    if sys.version_info < (3, 0):
+        file = io.open (pPath , mode='w', encoding='utf-8')
+        file.write ( pText.decode('unicode_escape') )
+        file.close()
+    else:
+        file = open (pPath, 'w', encoding='utf-8')
+        if pText is None:
+            pText = ""
+        file.write (pText)
+        file.close()
+
+#-------------------------------------------------------------------------------
+class noti_struc:
+    def __init__(self, json, text, html):
+        self.json = json
+        self.text = text
+        self.html = html        

pialert/initialise.py

@@ -0,0 +1,256 @@
+
+import os
+import time
+from pytz import timezone
+from cron_converter import Cron
+from pathlib import Path
+import datetime
+
+import conf 
+from const import fullConfPath
+from helper import collect_lang_strings, updateSubnets, initOrSetParam
+from logger import mylog
+from api import update_api
+from scheduler import schedule_class
+from plugin import get_plugins_configs, print_plugin_info
+
+#===============================================================================
+# Initialise user defined values
+#===============================================================================
+# We need access to the DB to save new values so need to define DB access methods first
+#-------------------------------------------------------------------------------
+
+#-------------------------------------------------------------------------------
+# Import user values
+# Check config dictionary
+def ccd(key, default, config_dir, name, inputtype, options, group, events=[], desc = "", regex = ""):
+    result = default
+
+    # use existing value if already supplied, otherwise default value is used
+    if key in config_dir:
+        result =  config_dir[key]
+
+    if inputtype == 'text':
+        result = result.replace('\'', "{s-quote}")
+
+    conf.mySettingsSQLsafe.append((key, name, desc, inputtype, options, regex, str(result), group, str(events)))
+    conf.mySettings.append((key, name, desc, inputtype, options, regex, result, group, str(events)))
+
+    return result
+#-------------------------------------------------------------------------------
+
+def importConfigs (db): 
+
+    sql = db.sql
+
+    # get config file name
+    config_file = Path(fullConfPath)
+
+    # Only import file if the file was modifed since last import.
+    # this avoids time zone issues as we just compare the previous timestamp to the current time stamp
+    mylog('debug', ['[Import Config] checking config file '])
+    mylog('debug', ['[Import Config] lastImportedConfFile     :', conf.lastImportedConfFile])
+    mylog('debug', ['[Import Config] file modified time       :', os.path.getmtime(config_file)])
+    
+
+    if (os.path.getmtime(config_file) == conf.lastImportedConfFile) :
+        mylog('debug', ['[Import Config] skipping config file import'])
+        return
+
+    conf.lastImportedConfFile = os.path.getmtime(config_file)  
+
+
+
+    
+    mylog('debug', ['[Import Config] importing config file'])
+    conf.mySettings = [] # reset settings
+    conf.mySettingsSQLsafe = [] # same as above but safe to be passed into a SQL query
+
+    c_d = read_config_file(config_file)
+
+    # Import setting if found in the dictionary
+    # General
+    conf.ENABLE_ARPSCAN = ccd('ENABLE_ARPSCAN', True , c_d, 'Enable arpscan', 'boolean', '', 'General', ['run']) 
+    conf.SCAN_SUBNETS = ccd('SCAN_SUBNETS', ['192.168.1.0/24 --interface=eth1', '192.168.1.0/24 --interface=eth0'] , c_d, 'Subnets to scan', 'subnets', '', 'General')    
+    conf.LOG_LEVEL = ccd('LOG_LEVEL', 'verbose' , c_d, 'Log verboseness', 'selecttext', "['none', 'minimal', 'verbose', 'debug']", 'General')
+    conf.TIMEZONE = ccd('TIMEZONE', 'Europe/Berlin' , c_d, 'Time zone', 'text', '', 'General')
+    conf.ENABLE_PLUGINS = ccd('ENABLE_PLUGINS', True , c_d, 'Enable plugins', 'boolean', '', 'General') 
+    conf.PIALERT_WEB_PROTECTION = ccd('PIALERT_WEB_PROTECTION', False , c_d, 'Enable logon', 'boolean', '', 'General')
+    conf.PIALERT_WEB_PASSWORD = ccd('PIALERT_WEB_PASSWORD', '8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92' , c_d, 'Logon password', 'readonly', '', 'General')
+    conf.INCLUDED_SECTIONS = ccd('INCLUDED_SECTIONS', ['internet', 'new_devices', 'down_devices', 'events', 'ports']   , c_d, 'Notify on', 'multiselect', "['internet', 'new_devices', 'down_devices', 'events', 'ports', 'plugins']", 'General')
+    conf.SCAN_CYCLE_MINUTES = ccd('SCAN_CYCLE_MINUTES', 5 , c_d, 'Scan cycle delay (m)', 'integer', '', 'General')
+    conf.DAYS_TO_KEEP_EVENTS = ccd('DAYS_TO_KEEP_EVENTS', 90 , c_d, 'Delete events days', 'integer', '', 'General')
+    conf.REPORT_DASHBOARD_URL = ccd('REPORT_DASHBOARD_URL', 'http://pi.alert/' , c_d, 'PiAlert URL', 'text', '', 'General')
+    conf.DIG_GET_IP_ARG = ccd('DIG_GET_IP_ARG', '-4 myip.opendns.com @resolver1.opendns.com' , c_d, 'DIG arguments', 'text', '', 'General')
+    conf.UI_LANG = ccd('UI_LANG', 'English' , c_d, 'Language Interface', 'selecttext', "['English', 'German', 'Spanish']", 'General')
+    conf.UI_PRESENCE = ccd('UI_PRESENCE', ['online', 'offline', 'archived']   , c_d, 'Include in presence', 'multiselect', "['online', 'offline', 'archived']", 'General')    
+
+    # Email
+    conf.REPORT_MAIL = ccd('REPORT_MAIL', False , c_d, 'Enable email', 'boolean', '', 'Email', ['test'])
+    conf.SMTP_SERVER = ccd('SMTP_SERVER', '' , c_d,'SMTP server URL', 'text', '', 'Email')
+    conf.SMTP_PORT = ccd('SMTP_PORT', 587 , c_d, 'SMTP port', 'integer', '', 'Email')
+    conf.REPORT_TO = ccd('REPORT_TO', 'user@gmail.com' , c_d, 'Email to', 'text', '', 'Email')
+    conf.REPORT_FROM = ccd('REPORT_FROM', 'Pi.Alert <user@gmail.com>' , c_d, 'Email Subject', 'text', '', 'Email')
+    conf.SMTP_SKIP_LOGIN = ccd('SMTP_SKIP_LOGIN', False , c_d, 'SMTP skip login', 'boolean', '', 'Email')
+    conf.SMTP_USER = ccd('SMTP_USER', '' , c_d, 'SMTP user', 'text', '', 'Email')
+    conf.SMTP_PASS = ccd('SMTP_PASS', '' , c_d, 'SMTP password', 'password', '', 'Email')
+    conf.SMTP_SKIP_TLS = ccd('SMTP_SKIP_TLS', False , c_d, 'SMTP skip TLS', 'boolean', '', 'Email')
+    conf.SMTP_FORCE_SSL = ccd('SMTP_FORCE_SSL', False , c_d, 'Force SSL', 'boolean', '', 'Email')
+
+    # Webhooks
+    conf.REPORT_WEBHOOK = ccd('REPORT_WEBHOOK', False , c_d, 'Enable Webhooks', 'boolean', '', 'Webhooks', ['test'])
+    conf.WEBHOOK_URL = ccd('WEBHOOK_URL', '' , c_d, 'Target URL', 'text', '', 'Webhooks')
+    conf.WEBHOOK_PAYLOAD = ccd('WEBHOOK_PAYLOAD', 'json' , c_d, 'Payload type', 'selecttext', "['json', 'html', 'text']", 'Webhooks')
+    conf.WEBHOOK_REQUEST_METHOD = ccd('WEBHOOK_REQUEST_METHOD', 'GET' , c_d, 'Req type', 'selecttext', "['GET', 'POST', 'PUT']", 'Webhooks')
+
+    # Apprise
+    conf.REPORT_APPRISE = ccd('REPORT_APPRISE', False , c_d, 'Enable Apprise', 'boolean', '', 'Apprise', ['test'])
+    conf.APPRISE_HOST = ccd('APPRISE_HOST', '' , c_d, 'Apprise host URL', 'text', '', 'Apprise')
+    conf.APPRISE_URL = ccd('APPRISE_URL', '' , c_d, 'Apprise notification URL', 'text', '', 'Apprise')
+    conf.APPRISE_PAYLOAD = ccd('APPRISE_PAYLOAD', 'html' , c_d, 'Payload type', 'selecttext', "['html', 'text']", 'Apprise')
+
+    # NTFY
+    conf.REPORT_NTFY = ccd('REPORT_NTFY', False , c_d, 'Enable NTFY', 'boolean', '', 'NTFY', ['test'])
+    conf.NTFY_HOST = ccd('NTFY_HOST', 'https://ntfy.sh' , c_d, 'NTFY host URL', 'text', '', 'NTFY')
+    conf.NTFY_TOPIC = ccd('NTFY_TOPIC', '' , c_d, 'NTFY topic', 'text', '', 'NTFY')
+    conf.NTFY_USER = ccd('NTFY_USER', '' , c_d, 'NTFY user', 'text', '', 'NTFY')
+    conf.NTFY_PASSWORD = ccd('NTFY_PASSWORD', '' , c_d, 'NTFY password', 'password', '', 'NTFY')
+
+    # PUSHSAFER
+    conf.REPORT_PUSHSAFER = ccd('REPORT_PUSHSAFER', False , c_d, 'Enable PUSHSAFER', 'boolean', '', 'PUSHSAFER', ['test'])
+    conf.PUSHSAFER_TOKEN = ccd('PUSHSAFER_TOKEN', 'ApiKey' , c_d, 'PUSHSAFER token', 'text', '', 'PUSHSAFER')
+
+    # MQTT
+    conf.REPORT_MQTT = ccd('REPORT_MQTT', False , c_d, 'Enable MQTT', 'boolean', '', 'MQTT')
+    conf.MQTT_BROKER = ccd('MQTT_BROKER', '' , c_d, 'MQTT broker', 'text', '', 'MQTT')
+    conf.MQTT_PORT = ccd('MQTT_PORT', 1883 , c_d, 'MQTT broker port', 'integer', '', 'MQTT')
+    conf.MQTT_USER = ccd('MQTT_USER', '' , c_d, 'MQTT user', 'text', '', 'MQTT')
+    conf.MQTT_PASSWORD = ccd('MQTT_PASSWORD', '' , c_d, 'MQTT password', 'password', '', 'MQTT')
+    conf.MQTT_QOS = ccd('MQTT_QOS', 0 , c_d, 'MQTT Quality of Service', 'selectinteger', "['0', '1', '2']", 'MQTT')
+    conf.MQTT_DELAY_SEC = ccd('MQTT_DELAY_SEC', 2 , c_d, 'MQTT delay', 'selectinteger', "['2', '3', '4', '5']", 'MQTT')
+
+    # DynDNS
+    conf.DDNS_ACTIVE = ccd('DDNS_ACTIVE', False , c_d, 'Enable DynDNS', 'boolean', '', 'DynDNS')
+    conf.DDNS_DOMAIN = ccd('DDNS_DOMAIN', 'your_domain.freeddns.org' , c_d, 'DynDNS domain URL', 'text', '', 'DynDNS')
+    conf.DDNS_USER = ccd('DDNS_USER', 'dynu_user' , c_d, 'DynDNS user', 'text', '', 'DynDNS')
+    conf.DDNS_PASSWORD = ccd('DDNS_PASSWORD', 'A0000000B0000000C0000000D0000000' , c_d, 'DynDNS password', 'password', '', 'DynDNS')
+    conf.DDNS_UPDATE_URL = ccd('DDNS_UPDATE_URL', 'https://api.dynu.com/nic/update?' , c_d, 'DynDNS update URL', 'text', '', 'DynDNS')
+
+    # PiHole
+    conf.PIHOLE_ACTIVE = ccd('PIHOLE_ACTIVE',  False, c_d, 'Enable PiHole mapping', 'boolean', '', 'PiHole')
+    conf.DHCP_ACTIVE = ccd('DHCP_ACTIVE', False , c_d, 'Enable PiHole DHCP', 'boolean', '', 'PiHole')
+
+    # PHOLUS
+    conf.PHOLUS_ACTIVE = ccd('PHOLUS_ACTIVE', False , c_d, 'Enable Pholus scans', 'boolean', '', 'Pholus')
+    conf.PHOLUS_TIMEOUT = ccd('PHOLUS_TIMEOUT', 20 , c_d, 'Pholus timeout', 'integer', '', 'Pholus')
+    conf.PHOLUS_FORCE = ccd('PHOLUS_FORCE', False , c_d, 'Pholus force check', 'boolean', '', 'Pholus')
+    conf.PHOLUS_RUN = ccd('PHOLUS_RUN', 'once' , c_d, 'Pholus enable schedule', 'selecttext', "['none', 'once', 'schedule']", 'Pholus')
+    conf.PHOLUS_RUN_TIMEOUT = ccd('PHOLUS_RUN_TIMEOUT', 600 , c_d, 'Pholus timeout schedule', 'integer', '', 'Pholus')   
+    conf.PHOLUS_RUN_SCHD = ccd('PHOLUS_RUN_SCHD', '0 4 * * *' , c_d, 'Pholus schedule', 'text', '', 'Pholus')
+    conf.PHOLUS_DAYS_DATA = ccd('PHOLUS_DAYS_DATA', 0 , c_d, 'Pholus keep days', 'integer', '', 'Pholus')
+    
+    # Nmap
+    conf.NMAP_ACTIVE = ccd('NMAP_ACTIVE', True , c_d, 'Enable Nmap scans', 'boolean', '', 'Nmap')
+    conf.NMAP_TIMEOUT = ccd('NMAP_TIMEOUT', 150 , c_d, 'Nmap timeout', 'integer', '', 'Nmap')
+    conf.NMAP_RUN = ccd('NMAP_RUN', 'none' , c_d, 'Nmap enable schedule', 'selecttext', "['none', 'once', 'schedule']", 'Nmap')
+    conf.NMAP_RUN_SCHD = ccd('NMAP_RUN_SCHD', '0 2 * * *' , c_d, 'Nmap schedule', 'text', '', 'Nmap')
+    conf.NMAP_ARGS = ccd('NMAP_ARGS', '-p -10000' , c_d, 'Nmap custom arguments', 'text', '', 'Nmap')
+
+    # API     
+    conf.API_CUSTOM_SQL = ccd('API_CUSTOM_SQL', 'SELECT * FROM Devices WHERE dev_PresentLastScan = 0' , c_d, 'Custom endpoint', 'text', '', 'API')
+
+    #  Init timezone in case it changed
+    conf.tz = timezone(conf.TIMEZONE) 
+    
+    # global mySchedules
+    # reset schedules
+    conf.mySchedules = [] 
+
+    # init pholus schedule
+    pholusSchedule = Cron(conf.PHOLUS_RUN_SCHD).schedule(start_date=datetime.datetime.now(conf.tz))    
+    
+    conf.mySchedules.append(schedule_class("pholus", pholusSchedule, pholusSchedule.next(), False))
+
+    # init nmap schedule
+    nmapSchedule = Cron(conf.NMAP_RUN_SCHD).schedule(start_date=datetime.datetime.now(conf.tz))
+    conf.mySchedules.append(schedule_class("nmap", nmapSchedule, nmapSchedule.next(), False))
+
+    # Format and prepare the list of subnets
+    conf.userSubnets = updateSubnets(conf.SCAN_SUBNETS)
+
+    # Plugins START
+    # -----------------
+    if conf.ENABLE_PLUGINS:
+        conf.plugins = get_plugins_configs()
+
+        mylog('none', ['[Config] Plugins: Number of dynamically loaded plugins: ', len(conf.plugins)])
+
+        #  handle plugins
+        for plugin in conf.plugins:
+            pref = plugin["unique_prefix"]  
+            print_plugin_info(plugin, ['display_name','description'])
+
+            # if plugin["enabled"] == 'true': 
+            
+            # collect plugin level language strings
+            collect_lang_strings(db, plugin, pref)
+            
+            for set in plugin["settings"]:
+                setFunction = set["function"]
+                # Setting code name / key  
+                key = pref + "_" + setFunction 
+
+                v = ccd(key, set["default_value"], c_d, set["name"][0]["string"], set["type"] , str(set["options"]), pref)
+
+                # Save the user defined value into the object
+                set["value"] = v
+
+                # Setup schedules
+                if setFunction == 'RUN_SCHD':
+                    newSchedule = Cron(v).schedule(start_date=datetime.datetime.now(conf.tz))
+                    conf.mySchedules.append(schedule_class(pref, newSchedule, newSchedule.next(), False))
+
+                # Collect settings related language strings
+                collect_lang_strings(db, set,  pref + "_" + set["function"])
+
+        conf.plugins_once_run = False
+    # -----------------
+    # Plugins END
+
+    
+           
+    
+
+    # Insert settings into the DB    
+    sql.execute ("DELETE FROM Settings")    
+    sql.executemany ("""INSERT INTO Settings ("Code_Name", "Display_Name", "Description", "Type", "Options",
+         "RegEx", "Value", "Group", "Events" ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", conf.mySettingsSQLsafe)
+
+    # Used to determine the next import
+    conf.lastTimeImported = time.time()
+
+    # Is used to display a message in the UI when old (outdated) settings are loaded    
+    initOrSetParam(db, "Back_Settings_Imported",(round(time.time() * 1000),) )    
+    
+    #commitDB(sql_connection)
+    db.commitDB()
+
+    #  update only the settings datasource
+    update_api(db, False, ["settings"])  
+    #TO DO this creates a circular reference between API and HELPER !
+
+    mylog('info', '[Config] Imported new config')
+
+
+
+#-------------------------------------------------------------------------------
+def read_config_file(filename):
+    """
+    retuns dict on the config file key:value pairs
+    """
+    mylog('info', '[Config] reading config file')
+    # load the variables from  pialert.conf
+    code = compile(filename.read_text(), filename.name, "exec")
+    confDict = {} # config dictionary
+    exec(code, {"__builtins__": {}}, confDict)
+    return confDict 

pialert/logger.py

@@ -0,0 +1,97 @@
+""" Colection of functions to support all logging for Pi.Alert """
+import sys
+import io
+import datetime
+
+import conf
+from const import *
+
+#-------------------------------------------------------------------------------
+# duplication from helper to avoid circle
+#-------------------------------------------------------------------------------
+def timeNow():
+        return datetime.datetime.now().replace(microsecond=0)
+
+
+#-------------------------------------------------------------------------------
+debugLevels =   [
+                    ('none', 0), ('minimal', 1), ('verbose', 2), ('debug', 3)
+                ]
+
+def mylog(requestedDebugLevel, n):
+
+    setLvl = 0  
+    reqLvl = 0  
+
+    #  Get debug urgency/relative weight
+    for lvl in debugLevels:
+        if conf.LOG_LEVEL == lvl[0]:
+            setLvl = lvl[1]
+        if requestedDebugLevel == lvl[0]:
+            reqLvl = lvl[1]
+
+    if reqLvl <= setLvl:
+        file_print (*n)
+
+#-------------------------------------------------------------------------------
+def file_print (*args):
+
+    result = timeNow().strftime ('%H:%M:%S') + ' '    
+       
+    for arg in args:                
+        result += str(arg)
+    print(result)
+
+    file = open(logPath + "/pialert.log", "a") 
+    file.write(result + '\n')
+    file.close()
+
+#-------------------------------------------------------------------------------
+def print_log (pText):
+
+    # Check LOG actived
+    if not conf.LOG_LEVEL == 'debug' :
+        return
+
+    # Current Time    
+    log_timestamp2 = datetime.datetime.now(conf.tz).replace(microsecond=0)
+
+    # Print line + time + elapsed time + text
+    file_print ('[LOG_LEVEL=debug] ',
+        # log_timestamp2, ' ',
+        log_timestamp2.strftime ('%H:%M:%S'), ' ',
+        pText)
+    
+
+    # Save current time to calculate elapsed time until next log
+    conf.log_timestamp = log_timestamp2 
+
+    return pText
+
+
+
+#-------------------------------------------------------------------------------
+def append_file_binary (pPath, input):    
+    file = open (pPath, 'ab') 
+    file.write (input) 
+    file.close() 
+
+
+#-------------------------------------------------------------------------------
+def logResult (stdout, stderr):
+    if stderr != None:
+        append_file_binary (logPath + '/stderr.log', stderr)
+    if stdout != None:
+        append_file_binary (logPath + '/stdout.log', stdout)  
+
+#-------------------------------------------------------------------------------
+def append_line_to_file (pPath, pText):
+    # append the line depending using the correct python version
+    if sys.version_info < (3, 0):
+        file = io.open (pPath , mode='a', encoding='utf-8')
+        file.write ( pText.decode('unicode_escape') ) 
+        file.close() 
+    else:
+        file = open (pPath, 'a', encoding='utf-8') 
+        file.write (pText) 
+        file.close() 

pialert/mac_vendor.py

@@ -0,0 +1,102 @@
+
+import subprocess
+
+from const import pialertPath, vendorsDB
+from helper import timeNow, updateState
+from logger import mylog
+
+
+#===============================================================================
+# UPDATE DEVICE MAC VENDORS
+#===============================================================================
+
+
+
+def update_devices_MAC_vendors (db, pArg = ''):
+    sql = db.sql # TO-DO
+    # Header    
+    updateState(db,"Upkeep: Vendors")
+    mylog('verbose', ['[', timeNow(), '] Upkeep - Update HW Vendors:' ])
+
+    # Update vendors DB (iab oui)
+    mylog('verbose', ['    Updating vendors DB (iab & oui)'])    
+    update_args = ['sh', pialertPath + '/update_vendors.sh', pArg]
+
+    try:
+        # try runnning a subprocess
+        update_output = subprocess.check_output (update_args)
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', ['    FAILED: Updating vendors DB, set LOG_LEVEL=debug for more info'])  
+        mylog('none', [e.output])        
+
+    # Initialize variables
+    recordsToUpdate = []
+    ignored = 0
+    notFound = 0
+
+    # All devices loop
+    mylog('verbose', ['    Searching devices vendor'])    
+    for device in sql.execute ("""SELECT * FROM Devices
+                                  WHERE dev_Vendor = '(unknown)' 
+                                     OR dev_Vendor =''
+                                     OR dev_Vendor IS NULL""") :
+        # Search vendor in HW Vendors DB
+        vendor = query_MAC_vendor (device['dev_MAC'])
+        if vendor == -1 :
+            notFound += 1
+        elif vendor == -2 :
+            ignored += 1
+        else :
+            recordsToUpdate.append ([vendor, device['dev_MAC']])
+            
+    # Print log    
+    mylog('verbose', ["    Devices Ignored:  ", ignored])
+    mylog('verbose', ["    Vendors Not Found:", notFound])
+    mylog('verbose', ["    Vendors updated:  ", len(recordsToUpdate) ])
+
+
+    # update devices
+    sql.executemany ("UPDATE Devices SET dev_Vendor = ? WHERE dev_MAC = ? ",
+        recordsToUpdate )
+
+    # Commit DB
+    db.commitDB()
+
+    if len(recordsToUpdate) > 0:
+        return True
+    else:
+        return False
+
+#-------------------------------------------------------------------------------
+def query_MAC_vendor (pMAC):
+    try :
+        # BUGFIX #6 - Fix pMAC parameter as numbers
+        pMACstr = str(pMAC)
+        
+        # Check MAC parameter
+        mac = pMACstr.replace (':','')
+        if len(pMACstr) != 17 or len(mac) != 12 :
+            return -2
+
+        # Search vendor in HW Vendors DB
+        mac = mac[0:6]
+        grep_args = ['grep', '-i', mac, vendorsDB]
+        # Execute command
+        try:
+            # try runnning a subprocess
+            grep_output = subprocess.check_output (grep_args)
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', ["[Mac Vendor Check] Error: ", e.output])
+            grep_output = "       There was an error, check logs for details"
+
+        # Return Vendor
+        vendor = grep_output[7:]
+        vendor = vendor.rstrip()
+        return vendor
+
+    # not Found
+    except subprocess.CalledProcessError :
+        return -1
+  

pialert/networkscan.py

@@ -0,0 +1,327 @@
+
+
+import conf
+from scanners.arpscan import execute_arpscan
+from scanners.pihole import copy_pihole_network, read_DHCP_leases
+from database import insertOnlineHistory
+from device import create_new_devices, print_scan_stats, save_scanned_devices, update_devices_data_from_scan, update_devices_names
+from helper import timeNow
+from logger import mylog
+from reporting import skip_repeated_notifications
+
+
+
+#===============================================================================
+# SCAN NETWORK
+#===============================================================================
+
+
+def scan_network (db):
+    sql = db.sql #TO-DO
+
+
+    # Header
+    # moved updateState to main loop
+    # updateState(db,"Scan: Network")
+    mylog('verbose', ['[Network Scan] Scan Devices:' ])       
+
+    # Query ScanCycle properties    
+    scanCycle_data = query_ScanCycle_Data (db, True)
+    if scanCycle_data is None:
+        mylog('none', ['\n'])        
+        mylog('none', ['[Network Scan]*************** ERROR ***************'])
+        mylog('none', ['[Network Scan] ScanCycle %s not found' % conf.cycle ])
+        mylog('none', ['[Network Scan]    Exiting...\n'])
+        return False
+
+    db.commitDB()
+
+    # arp-scan command
+    conf.arpscan_devices = []
+    if conf.ENABLE_ARPSCAN:    
+        mylog('verbose','[Network Scan] arp-scan start')    
+        conf.arpscan_devices = execute_arpscan (conf.userSubnets)
+        mylog('verbose','[Network Scan] arp-scan ends')
+
+    # Pi-hole method    
+    if conf.PIHOLE_ACTIVE :       
+        mylog('verbose','[Network Scan] Pi-hole start')        
+        copy_pihole_network(db) 
+        db.commitDB() 
+
+    # DHCP Leases method    
+    if conf.DHCP_ACTIVE :        
+        mylog('verbose','[Network Scan] DHCP Leases start')        
+        read_DHCP_leases (db) 
+        db.commitDB()
+
+
+
+def process_scan (db, arpscan_devices):
+
+
+    # Query ScanCycle properties    
+    scanCycle_data = query_ScanCycle_Data (db, True)
+    if scanCycle_data is None:
+        mylog('none', ['\n'])        
+        mylog('none', ['[Process Scan]*************** ERROR ***************'])
+        mylog('none', ['[Process Scan] ScanCycle %s not found' % conf.cycle ])
+        mylog('none', ['[Process Scan]    Exiting...\n'])
+        return False
+
+    db.commitDB()
+
+    # ScanCycle data        
+    cycle_interval  = scanCycle_data['cic_EveryXmin']
+
+    # Load current scan data
+    mylog('verbose','[Process Scan]  Processing scan results')     
+    save_scanned_devices (db, arpscan_devices, cycle_interval)    
+    
+    # Print stats
+    mylog('none','[Process Scan] Print Stats')
+    print_scan_stats(db)
+    mylog('none','[Process Scan] Stats end')
+
+    # Create Events
+    mylog('verbose','[Process Scan] Updating DB Info')
+    mylog('verbose','[Process Scan] Sessions Events (connect / discconnect)')
+    insert_events(db)
+
+    # Create New Devices
+    # after create events -> avoid 'connection' event
+    mylog('verbose','[Process Scan] Creating new devices')
+    create_new_devices (db)
+
+    # Update devices info
+    mylog('verbose','[Process Scan] Updating Devices Info')
+    update_devices_data_from_scan (db)
+
+    # Resolve devices names
+    mylog('verbose','[Process Scan] Resolve devices names')
+    update_devices_names(db)
+
+    # Void false connection - disconnections
+    mylog('verbose','[Process Scan] Voiding false (ghost) disconnections')    
+    void_ghost_disconnections (db)
+
+    # Pair session events (Connection / Disconnection)
+    mylog('verbose','[Process Scan] Pairing session events (connection / disconnection) ')
+    pair_sessions_events(db)  
+  
+    # Sessions snapshot
+    mylog('verbose','[Process Scan] Creating sessions snapshot')
+    create_sessions_snapshot (db)
+
+    # Sessions snapshot
+    mylog('verbose','[Process Scan] Inserting scan results into Online_History')
+    insertOnlineHistory(db)
+  
+    # Skip repeated notifications
+    mylog('verbose','[Process Scan] Skipping repeated notifications')
+    skip_repeated_notifications (db)
+  
+    # Commit changes    
+    db.commitDB()
+
+    # moved plugin execution to main loop
+    # if ENABLE_PLUGINS:
+    #    run_plugin_scripts(db,'always_after_scan')
+
+
+#-------------------------------------------------------------------------------
+def query_ScanCycle_Data (db, pOpenCloseDB = False, cycle = 1):
+    # Query Data
+    db.sql.execute ("""SELECT cic_arpscanCycles, cic_EveryXmin
+                    FROM ScanCycles
+                    WHERE cic_ID = ? """, (cycle,))
+    sqlRow = db.sql.fetchone()
+
+    # Return Row
+    return sqlRow
+
+
+
+#-------------------------------------------------------------------------------
+def void_ghost_disconnections (db):
+    sql = db.sql #TO-DO
+    startTime = timeNow()
+    # Void connect ghost events (disconnect event exists in last X min.) 
+    mylog('debug','[Void Ghost Con] - 1 Connect ghost events')
+    sql.execute ("""UPDATE Events SET eve_PairEventRowid = Null,
+                        eve_EventType ='VOIDED - ' || eve_EventType
+                    WHERE eve_MAC != 'Internet'
+                      AND eve_EventType = 'Connected'
+                      AND eve_DateTime = ?
+                      AND eve_MAC IN (
+                          SELECT Events.eve_MAC
+                          FROM CurrentScan, Devices, ScanCycles, Events 
+                          WHERE cur_ScanCycle = ?
+                            AND dev_MAC = cur_MAC
+                            AND dev_ScanCycle = cic_ID
+                            AND cic_ID = cur_ScanCycle
+                            AND eve_MAC = cur_MAC
+                            AND eve_EventType = 'Disconnected'
+                            AND eve_DateTime >=
+                                DATETIME (?, '-' || cic_EveryXmin ||' minutes')
+                          ) """,
+                    (startTime, conf.cycle, startTime)   )
+
+    # Void connect paired events
+    mylog('debug','[Void Ghost Con] - 2 Paired events')
+    sql.execute ("""UPDATE Events SET eve_PairEventRowid = Null 
+                    WHERE eve_MAC != 'Internet'
+                      AND eve_PairEventRowid IN (
+                          SELECT Events.RowID
+                          FROM CurrentScan, Devices, ScanCycles, Events 
+                          WHERE cur_ScanCycle = ?
+                            AND dev_MAC = cur_MAC
+                            AND dev_ScanCycle = cic_ID
+                            AND cic_ID = cur_ScanCycle
+                            AND eve_MAC = cur_MAC
+                            AND eve_EventType = 'Disconnected'
+                            AND eve_DateTime >=
+                                DATETIME (?, '-' || cic_EveryXmin ||' minutes')
+                          ) """,
+                    (conf.cycle, startTime)   )
+
+    # Void disconnect ghost events 
+    mylog('debug','[Void Ghost Con] - 3 Disconnect ghost events')
+    sql.execute ("""UPDATE Events SET eve_PairEventRowid = Null, 
+                        eve_EventType = 'VOIDED - '|| eve_EventType
+                    WHERE eve_MAC != 'Internet'
+                      AND ROWID IN (
+                          SELECT Events.RowID
+                          FROM CurrentScan, Devices, ScanCycles, Events 
+                          WHERE cur_ScanCycle = ?
+                            AND dev_MAC = cur_MAC
+                            AND dev_ScanCycle = cic_ID
+                            AND cic_ID = cur_ScanCycle
+                            AND eve_MAC = cur_MAC
+                            AND eve_EventType = 'Disconnected'
+                            AND eve_DateTime >=
+                                DATETIME (?, '-' || cic_EveryXmin ||' minutes')
+                          ) """,
+                    (conf.cycle, startTime)   )
+    mylog('debug','[Void Ghost Con] Void Ghost Connections end')
+    db.commitDB()
+
+#-------------------------------------------------------------------------------
+def pair_sessions_events (db):
+    sql = db.sql #TO-DO
+
+    # NOT NECESSARY FOR INCREMENTAL UPDATE
+    # print_log ('Pair session - 1 Clean')
+    # sql.execute ("""UPDATE Events
+    #                 SET eve_PairEventRowid = NULL
+    #                 WHERE eve_EventType IN ('New Device', 'Connected')
+    #              """ )
+    
+
+    # Pair Connection / New Device events
+    mylog('debug','[Pair Session] - 1 Connections / New Devices')
+    sql.execute ("""UPDATE Events
+                    SET eve_PairEventRowid =
+                       (SELECT ROWID
+                        FROM Events AS EVE2
+                        WHERE EVE2.eve_EventType IN ('New Device', 'Connected',
+                            'Device Down', 'Disconnected')
+                           AND EVE2.eve_MAC = Events.eve_MAC
+                           AND EVE2.eve_Datetime > Events.eve_DateTime
+                        ORDER BY EVE2.eve_DateTime ASC LIMIT 1)
+                    WHERE eve_EventType IN ('New Device', 'Connected')
+                    AND eve_PairEventRowid IS NULL
+                 """ )
+
+    # Pair Disconnection / Device Down
+    mylog('debug','[Pair Session] - 2 Disconnections')
+    sql.execute ("""UPDATE Events
+                    SET eve_PairEventRowid =
+                        (SELECT ROWID
+                         FROM Events AS EVE2
+                         WHERE EVE2.eve_PairEventRowid = Events.ROWID)
+                    WHERE eve_EventType IN ('Device Down', 'Disconnected')
+                      AND eve_PairEventRowid IS NULL
+                 """ )
+    mylog('debug','[Pair Session] Pair session end')
+
+    db.commitDB()
+
+#-------------------------------------------------------------------------------
+def create_sessions_snapshot (db):
+    sql = db.sql #TO-DO
+
+    # Clean sessions snapshot
+    mylog('debug','[Sessions Snapshot] - 1 Clean')
+    sql.execute ("DELETE FROM SESSIONS" )
+
+    # Insert sessions
+    mylog('debug','[Sessions Snapshot] - 2 Insert')
+    sql.execute ("""INSERT INTO Sessions
+                    SELECT * FROM Convert_Events_to_Sessions""" )
+
+    mylog('debug','[Sessions Snapshot] Sessions end')
+    db.commitDB()
+
+
+#-------------------------------------------------------------------------------
+def insert_events (db):
+    sql = db.sql #TO-DO
+    startTime = timeNow()    
+    
+    # Check device down
+    mylog('debug','[Events] - 1 - Devices down')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT dev_MAC, dev_LastIP, ?, 'Device Down', '', 1
+                    FROM Devices
+                    WHERE dev_AlertDeviceDown = 1
+                      AND dev_PresentLastScan = 1
+                      AND dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (startTime, conf.cycle) )
+
+    # Check new connections
+    mylog('debug','[Events] - 2 - New Connections')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT cur_MAC, cur_IP, ?, 'Connected', '', dev_AlertEvents
+                    FROM Devices, CurrentScan
+                    WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
+                      AND dev_PresentLastScan = 0
+                      AND dev_ScanCycle = ? """,
+                    (startTime, conf.cycle) )
+
+    # Check disconnections
+    mylog('debug','[Events] - 3 - Disconnections')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT dev_MAC, dev_LastIP, ?, 'Disconnected', '',
+                        dev_AlertEvents
+                    FROM Devices
+                    WHERE dev_AlertDeviceDown = 0
+                      AND dev_PresentLastScan = 1
+                      AND dev_ScanCycle = ?
+                      AND NOT EXISTS (SELECT 1 FROM CurrentScan
+                                      WHERE dev_MAC = cur_MAC
+                                        AND dev_ScanCycle = cur_ScanCycle) """,
+                    (startTime, conf.cycle) )
+
+    # Check IP Changed
+    mylog('debug','[Events] - 4 - IP Changes')
+    sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    SELECT cur_MAC, cur_IP, ?, 'IP Changed',
+                        'Previous IP: '|| dev_LastIP, dev_AlertEvents
+                    FROM Devices, CurrentScan
+                    WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
+                      AND dev_ScanCycle = ?
+                      AND dev_LastIP <> cur_IP """,
+                    (startTime, conf.cycle) )
+    mylog('debug','[Events] - Events end')

pialert/plugin.py

@@ -0,0 +1,588 @@
+import os
+import json
+import subprocess
+import datetime
+from collections import namedtuple
+
+# pialert modules
+import conf
+from const import pluginsPath, logPath
+from logger import mylog
+from helper import timeNow,  updateState, get_file_content, write_file
+from api import update_api
+
+#-------------------------------------------------------------------------------
+def run_plugin_scripts(db, runType):
+
+    # Header
+    updateState(db,"Run: Plugins")
+
+    mylog('debug', ['[Plugins] Check if any plugins need to be executed on run type: ', runType])
+
+    for plugin in conf.plugins:
+
+        shouldRun = False
+
+        set = get_plugin_setting(plugin, "RUN")
+        if set != None and set['value'] == runType:
+            if runType != "schedule":
+                shouldRun = True
+            elif  runType == "schedule":
+                # run if overdue scheduled time   
+                prefix = plugin["unique_prefix"]
+
+                #  check scheduels if any contains a unique plugin prefix matching the current plugin
+                for schd in conf.mySchedules:
+                    if schd.service == prefix:          
+                        # Check if schedule overdue
+                        shouldRun = schd.runScheduleCheck()  
+                        if shouldRun:
+                            # note the last time the scheduled plugin run was executed
+                            schd.last_run = timeNow()
+
+        if shouldRun:            
+                        
+            print_plugin_info(plugin, ['display_name'])
+            mylog('debug', ['[Plugins] CMD: ', get_plugin_setting(plugin, "CMD")["value"]])
+            execute_plugin(db, plugin) 
+
+
+
+
+
+#-------------------------------------------------------------------------------
+def get_plugins_configs():
+    
+    pluginsList = []
+
+    # only top level directories required. No need for the loop
+    # for root, dirs, files in os.walk(pluginsPath):
+    
+    dirs = next(os.walk(pluginsPath))[1]
+    for d in dirs:            # Loop over directories, not files    
+        if not d.startswith( "__" ):        # ignore __pycache__ 
+            pluginsList.append(json.loads(get_file_content(pluginsPath + "/" + d + '/config.json')))          
+
+    return pluginsList
+
+
+
+#-------------------------------------------------------------------------------
+def print_plugin_info(plugin, elements = ['display_name']):
+
+    mylog('verbose', ['[Plugins] ---------------------------------------------']) 
+
+    for el in elements:
+        res = get_plugin_string(plugin, el)
+        mylog('verbose', ['[Plugins] ', el ,': ', res]) 
+
+
+#-------------------------------------------------------------------------------
+# Gets the whole setting object
+def get_plugin_setting(plugin, function_key):
+    
+    result = None
+
+    for set in plugin['settings']:
+        if set["function"] == function_key:
+          result =  set 
+
+    if result == None:
+        mylog('none', ['[Plugins] Setting with "function":"', function_key, '" is missing in plugin: ', get_plugin_string(plugin, 'display_name')])
+
+    return result
+
+#-------------------------------------------------------------------------------
+#  Return whole setting touple
+def get_setting(key):
+    result = None
+    # index order: key, name, desc, inputtype, options, regex, result, group, events
+    for set in conf.mySettings:
+        if set[0] == key:
+            result = set
+    
+    if result is None:
+        mylog('info', [' Error - setting_missing - Setting not found for key: ', key])           
+        mylog('info', [' Error - logging the settings into file: ', logPath + '/setting_missing.json'])           
+        write_file (logPath + '/setting_missing.json', json.dumps({ 'data' : conf.mySettings}))    
+
+    return result
+        
+
+#-------------------------------------------------------------------------------
+# Get localized string value on the top JSON depth, not recursive
+def get_plugin_string(props, el):
+
+    result = ''
+
+    if el in props['localized']:
+        for val in props[el]:
+            if val['language_code'] == 'en_us':
+                result = val['string']
+        
+        if result == '':
+            result = 'en_us string missing'
+
+    else:
+        result = props[el]
+    
+    return result
+
+
+#-------------------------------------------------------------------------------
+# Executes the plugin command specified in the setting with the function specified as CMD 
+def execute_plugin(db, plugin):
+    sql = db.sql  
+
+    # ------- necessary settings check  --------
+    set = get_plugin_setting(plugin, "CMD")
+
+    #  handle missing "function":"CMD" setting
+    if set == None:                
+        return 
+
+    set_CMD = set["value"]
+
+    set = get_plugin_setting(plugin, "RUN_TIMEOUT")
+
+    #  handle missing "function":"<unique_prefix>_TIMEOUT" setting
+    if set == None:   
+        set_RUN_TIMEOUT = 10
+    else:     
+        set_RUN_TIMEOUT = set["value"] 
+
+    mylog('debug', ['[Plugins] Timeout: ', set_RUN_TIMEOUT])     
+
+    #  Prepare custom params
+    params = []
+
+    if "params" in plugin:
+        for param in plugin["params"]:            
+            resolved = ""
+
+            #  Get setting value
+            if param["type"] == "setting":
+                resolved = get_setting(param["value"])
+
+                if resolved != None:
+                    resolved = plugin_param_from_glob_set(resolved)
+
+            #  Get Sql result
+            if param["type"] == "sql":
+                resolved = flatten_array(db.get_sql_array(param["value"]))
+
+            if resolved == None:
+                mylog('none', ['[Plugins] The parameter "name":"', param["name"], '" was resolved as None'])
+
+            else:
+                params.append( [param["name"], resolved] )
+    
+
+    # build SQL query parameters to insert into the DB
+    sqlParams = []
+
+    # python-script 
+    if plugin['data_source'] == 'python-script':
+        # ------- prepare params --------
+        # prepare command from plugin settings, custom parameters  
+        command = resolve_wildcards_arr(set_CMD.split(), params)        
+
+        # Execute command
+        mylog('verbose', ['[Plugins] Executing: ', set_CMD])
+        mylog('debug',   ['[Plugins] Resolved : ', command])        
+
+        try:
+            # try runnning a subprocess with a forced timeout in case the subprocess hangs
+            output = subprocess.check_output (command, universal_newlines=True,  stderr=subprocess.STDOUT, timeout=(set_RUN_TIMEOUT))
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', [e.output])
+            mylog('none', ['[Plugins] Error - enable LOG_LEVEL=debug and check logs'])            
+        except subprocess.TimeoutExpired as timeErr:
+            mylog('none', ['[Plugins] TIMEOUT - the process forcefully terminated as timeout reached']) 
+
+
+        #  check the last run output
+        f = open(pluginsPath + '/' + plugin["code_name"] + '/last_result.log', 'r+')
+        newLines = f.read().split('\n')
+        f.close()        
+
+        # cleanup - select only lines containing a separator to filter out unnecessary data
+        newLines = list(filter(lambda x: '|' in x, newLines))  
+
+        # # regular logging
+        # for line in newLines:
+        #     append_line_to_file (pluginsPath + '/plugin.log', line +'\n')         
+        
+        for line in newLines:
+            columns = line.split("|")
+            # There has to be always 9 columns
+            if len(columns) == 9:
+                sqlParams.append((plugin["unique_prefix"], columns[0], columns[1], 'null', columns[2], columns[3], columns[4], columns[5], columns[6], 0, columns[7], 'null', columns[8]))
+            else:
+                mylog('none', ['[Plugins]: Skipped invalid line in the output: ', line])
+    
+    # pialert-db-query
+    if plugin['data_source'] == 'pialert-db-query':
+        # replace single quotes wildcards
+        q = set_CMD.replace("{s-quote}", '\'')
+
+        # Execute command
+        mylog('verbose', ['[Plugins] Executing: ', q])
+
+        # set_CMD should contain a SQL query        
+        arr = db.get_sql_array (q) 
+
+        for row in arr:
+            # There has to be always 9 columns
+            if len(row) == 9 and (row[0] in ['','null']) == False :
+                sqlParams.append((plugin["unique_prefix"], row[0], handle_empty(row[1]), 'null', row[2], row[3], row[4], handle_empty(row[5]), handle_empty(row[6]), 0, row[7], 'null', row[8]))
+            else:
+                mylog('none', ['[Plugins]: Skipped invalid sql result'])
+
+
+    # check if the subprocess / SQL query failed / there was no valid output
+    if len(sqlParams) == 0: 
+        mylog('none', ['[Plugins] No output received from the plugin ', plugin["unique_prefix"], ' - enable LOG_LEVEL=debug and check logs'])
+        return  
+    else: 
+        mylog('verbose', ['[Plugins]: SUCCESS, received ', len(sqlParams), ' entries'])  
+
+    # process results if any
+    if len(sqlParams) > 0:                
+        sql.executemany ("""INSERT INTO Plugins_Events ("Plugin", "Object_PrimaryID", "Object_SecondaryID", "DateTimeCreated", "DateTimeChanged", "Watched_Value1", "Watched_Value2", "Watched_Value3", "Watched_Value4", "Status" ,"Extra", "UserData", "ForeignKey") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", sqlParams) 
+        db.commitDB()
+        sql.executemany ("""INSERT INTO Plugins_History ("Plugin", "Object_PrimaryID", "Object_SecondaryID", "DateTimeCreated", "DateTimeChanged", "Watched_Value1", "Watched_Value2", "Watched_Value3", "Watched_Value4", "Status" ,"Extra", "UserData", "ForeignKey") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", sqlParams) 
+        db.commitDB()
+
+        process_plugin_events(db, plugin)
+
+        # update API endpoints
+        update_api(db, False, ["plugins_events","plugins_objects"])   
+
+#-------------------------------------------------------------------------------
+def custom_plugin_decoder(pluginDict):
+    return namedtuple('X', pluginDict.keys())(*pluginDict.values())        
+
+#-------------------------------------------------------------------------------
+# Handle empty value 
+def handle_empty(value):    
+    if value == '' or value is None:
+        value = 'null'
+
+    return value    
+
+
+#-------------------------------------------------------------------------------
+# Flattens a setting to make it passable to a script
+def plugin_param_from_glob_set(globalSetting):
+
+    setVal = globalSetting[6] # setting value
+    setTyp = globalSetting[3] # setting type
+
+
+    noConversion = ['text', 'integer', 'boolean', 'password', 'readonly', 'selectinteger', 'selecttext' ]
+    arrayConversion = ['multiselect', 'list'] 
+
+    if setTyp in noConversion:
+        return setVal
+
+    if setTyp in arrayConversion:
+        return flatten_array(setVal)
+
+#-------------------------------------------------------------------------------
+# Gets the setting value
+def get_plugin_setting_value(plugin, function_key):
+    
+    resultObj = get_plugin_setting(plugin, function_key)
+
+    if resultObj != None:
+        return resultObj["value"]
+
+    return None
+
+#-------------------------------------------------------------------------------
+#  Return setting value
+def get_setting_value(key):
+    
+    set = get_setting(key)
+
+    if get_setting(key) is not None:
+
+        setVal = set[6] # setting value
+        setTyp = set[3] # setting type
+
+        return setVal
+
+    return ''
+
+#-------------------------------------------------------------------------------
+def flatten_array(arr):
+    
+    tmp = ''
+
+    mylog('debug', arr)
+
+    for arrayItem in arr:        
+        # only one column flattening is supported
+        if isinstance(arrayItem, list):            
+            arrayItem = str(arrayItem[0])
+
+        tmp += arrayItem + ','
+        # tmp = tmp.replace("'","").replace(' ','') # No single quotes or empty spaces allowed
+        tmp = tmp.replace("'","") # No single quotes allowed
+
+    return tmp[:-1] # Remove last comma ','
+
+
+#-------------------------------------------------------------------------------
+# Replace {wildcars} with parameters
+def resolve_wildcards_arr(commandArr, params):
+
+    mylog('debug', ['[Plugins]: Pre-Resolved CMD: '] + commandArr)   
+
+    for param in params:
+        # mylog('debug', ['[Plugins]: key     : {', param[0], '}'])
+        # mylog('debug', ['[Plugins]: resolved: ', param[1]])
+
+        i = 0
+        
+        for comPart in commandArr:
+
+            commandArr[i] = comPart.replace('{' + param[0] + '}', param[1]).replace('{s-quote}',"'")
+
+            i += 1
+
+    return commandArr    
+
+
+#-------------------------------------------------------------------------------
+# Combine plugin objects, keep user-defined values, created time, changed time if nothing changed and the index
+def combine_plugin_objects(old, new):    
+    
+    new.userData = old.userData 
+    new.index = old.index 
+    new.created = old.created 
+
+    # Keep changed time if nothing changed
+    if new.status in ['watched-not-changed']:
+        new.changed = old.changed
+
+    #  return the new object, with some of the old values
+    return new
+
+#-------------------------------------------------------------------------------
+# Check if watched values changed for the given plugin
+def process_plugin_events(db, plugin):    
+    sql = db.sql
+    
+    ##global pluginObjects, pluginEvents
+
+    pluginPref = plugin["unique_prefix"]
+
+    mylog('debug', ['[Plugins] Processing        : ', pluginPref])
+
+    plugObjectsArr = db.get_sql_array ("SELECT * FROM Plugins_Objects where Plugin = '" + str(pluginPref)+"'") 
+    plugEventsArr  = db.get_sql_array ("SELECT * FROM Plugins_Events where Plugin = '" + str(pluginPref)+"'") 
+
+    pluginObjects = []
+    pluginEvents  = []
+
+    for obj in plugObjectsArr: 
+        pluginObjects.append(plugin_object_class(plugin, obj))
+
+    existingPluginObjectsCount = len(pluginObjects)
+
+    mylog('debug', ['[Plugins] Existing objects        : ', existingPluginObjectsCount])
+    mylog('debug', ['[Plugins] New and existing events : ', len(plugEventsArr)])
+
+    # set status as new - will be changed later if conditions are fulfilled, e.g. entry found
+    for eve in plugEventsArr:
+        tmpObject = plugin_object_class(plugin, eve)        
+        tmpObject.status = "new"
+        pluginEvents.append(tmpObject)
+
+    
+    #  Update the status to "exists"
+    index = 0
+    for tmpObjFromEvent in pluginEvents:        
+
+        #  compare hash of the IDs for uniqueness
+        if any(x.idsHash == tmpObject.idsHash for x in pluginObjects):
+            mylog('debug', ['[Plugins] Found existing object'])
+            pluginEvents[index].status = "exists"            
+        index += 1
+
+    # Loop thru events and update the one that exist to determine if watched columns changed
+    index = 0
+    for tmpObjFromEvent in pluginEvents:  
+
+        if tmpObjFromEvent.status == "exists": 
+
+            #  compare hash of the changed watched columns for uniqueness
+            if any(x.watchedHash != tmpObject.watchedHash for x in pluginObjects):
+                pluginEvents[index].status = "watched-changed"                            
+            else:
+                pluginEvents[index].status = "watched-not-changed"  
+        index += 1
+
+    # Merge existing plugin objects with newly discovered ones and update existing ones with new values
+    for eveObj in pluginEvents:
+        if eveObj.status == 'new':
+            pluginObjects.append(eveObj)
+        else:
+            index = 0
+            for plugObj in pluginObjects:
+                # find corresponding object for the event and merge
+                if plugObj.idsHash == eveObj.idsHash:               
+                    pluginObjects[index] =  combine_plugin_objects(plugObj, eveObj)                    
+
+                index += 1
+
+    # Update the DB
+    # ----------------------------
+    # Update the Plugin_Objects    
+    for plugObj in pluginObjects: 
+
+        createdTime = plugObj.created
+
+        if plugObj.status == 'new':
+            
+            createdTime = plugObj.changed
+
+            sql.execute ("INSERT INTO Plugins_Objects (Plugin, Object_PrimaryID, Object_SecondaryID, DateTimeCreated, DateTimeChanged, Watched_Value1, Watched_Value2, Watched_Value3, Watched_Value4, Status, Extra, UserData, ForeignKey) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)", (plugObj.pluginPref, plugObj.primaryId , plugObj.secondaryId , createdTime, plugObj.changed , plugObj.watched1 , plugObj.watched2 , plugObj.watched3 , plugObj.watched4 , plugObj.status , plugObj.extra, plugObj.userData, plugObj.foreignKey ))    
+        else:
+            sql.execute (f"UPDATE Plugins_Objects set Plugin = '{plugObj.pluginPref}', DateTimeChanged = '{plugObj.changed}', Watched_Value1 = '{plugObj.watched1}', Watched_Value2 = '{plugObj.watched2}', Watched_Value3 = '{plugObj.watched3}', Watched_Value4 = '{plugObj.watched4}', Status = '{plugObj.status}', Extra = '{plugObj.extra}', ForeignKey = '{plugObj.foreignKey}' WHERE \"Index\" = {plugObj.index}")            
+
+    # Update the Plugins_Events with the new statuses    
+    sql.execute (f'DELETE FROM Plugins_Events where Plugin = "{pluginPref}"')
+
+    for plugObj in pluginEvents: 
+
+        createdTime = plugObj.created
+
+        # use the same datetime for created and changed if a new entry
+        if plugObj.status == 'new':
+            createdTime = plugObj.changed  
+
+        #  insert only events if they are to be reported on
+        if plugObj.status in get_plugin_setting_value(plugin, "REPORT_ON"):  
+
+            sql.execute ("INSERT INTO Plugins_Events (Plugin, Object_PrimaryID, Object_SecondaryID, DateTimeCreated, DateTimeChanged, Watched_Value1, Watched_Value2, Watched_Value3, Watched_Value4, Status,  Extra, UserData, ForeignKey) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)", (plugObj.pluginPref, plugObj.primaryId , plugObj.secondaryId , createdTime, plugObj.changed , plugObj.watched1 , plugObj.watched2 , plugObj.watched3 , plugObj.watched4 , plugObj.status , plugObj.extra, plugObj.userData, plugObj.foreignKey ))
+
+    # Perform databse table mapping if enabled for the plugin   
+    if len(pluginEvents) > 0 and  "mapped_to_table" in plugin:
+
+        sqlParams = []        
+
+        dbTable = plugin['mapped_to_table']
+
+        mylog('debug', ['[Plugins] Mapping objects to database table: ', dbTable])
+
+        #  collect all columns to be mapped
+        mappedCols = []        
+        columnsStr = ''
+        valuesStr = ''
+
+        for clmn in plugin['database_column_definitions']:            
+            if 'mapped_to_column' in clmn:                
+                mappedCols.append(clmn)
+                columnsStr = f'{columnsStr}, "{clmn["mapped_to_column"]}"'
+                valuesStr = f'{valuesStr}, ?'
+
+        if len(columnsStr) > 0:
+            columnsStr = columnsStr[1:] # remove first ','
+            valuesStr = valuesStr[1:] # remove first ','
+
+        # map the column names to plugin object event values
+        for plgEv in pluginEvents:
+
+            tmpList = []
+
+            for col in mappedCols:
+                if col['column'] == 'Index':
+                    tmpList.append(plgEv.index)
+                elif col['column'] == 'Plugin':
+                    tmpList.append(plgEv.pluginPref)
+                elif col['column'] == 'Object_PrimaryID':
+                    tmpList.append(plgEv.primaryId)
+                elif col['column'] == 'Object_SecondaryID':
+                    tmpList.append(plgEv.secondaryId)
+                elif col['column'] == 'DateTimeCreated':
+                    tmpList.append(plgEv.created)
+                elif col['column'] == 'DateTimeChanged':
+                    tmpList.append(plgEv.changed)
+                elif col['column'] == 'Watched_Value1':
+                    tmpList.append(plgEv.watched1)
+                elif col['column'] == 'Watched_Value2':
+                    tmpList.append(plgEv.watched2)
+                elif col['column'] == 'Watched_Value3':
+                    tmpList.append(plgEv.watched3)
+                elif col['column'] == 'Watched_Value4':
+                    tmpList.append(plgEv.watched4)
+                elif col['column'] == 'UserData':
+                    tmpList.append(plgEv.userData)
+                elif col['column'] == 'Extra':
+                    tmpList.append(plgEv.extra)
+                elif col['column'] == 'Status':
+                    tmpList.append(plgEv.status)               
+                
+            sqlParams.append(tuple(tmpList))
+
+        q = f'INSERT into {dbTable} ({columnsStr}) VALUES ({valuesStr})'
+
+        mylog('debug', ['[Plugins] SQL query for mapping: ', q ])
+
+        sql.executemany (q, sqlParams) 
+
+    db.commitDB()
+
+
+
+
+
+#-------------------------------------------------------------------------------
+class plugin_object_class:
+    def __init__(self, plugin, objDbRow):
+        self.index        = objDbRow[0]
+        self.pluginPref   = objDbRow[1]
+        self.primaryId    = objDbRow[2]
+        self.secondaryId  = objDbRow[3]
+        self.created      = objDbRow[4]
+        self.changed      = objDbRow[5]
+        self.watched1     = objDbRow[6]
+        self.watched2     = objDbRow[7]
+        self.watched3     = objDbRow[8]
+        self.watched4     = objDbRow[9]
+        self.status       = objDbRow[10]
+        self.extra        = objDbRow[11]
+        self.userData     = objDbRow[12]
+        self.foreignKey   = objDbRow[13]
+
+        # self.idsHash      = str(hash(str(self.primaryId) + str(self.secondaryId)))    
+        self.idsHash      = str(self.primaryId) + str(self.secondaryId)
+
+        self.watchedClmns = []
+        self.watchedIndxs = []          
+
+        setObj = get_plugin_setting(plugin, 'WATCH')
+
+        indexNameColumnMapping = [(6, 'Watched_Value1' ), (7, 'Watched_Value2' ), (8, 'Watched_Value3' ), (9, 'Watched_Value4' )]
+
+        if setObj is not None:            
+            
+            self.watchedClmns = setObj["value"]
+
+            for clmName in self.watchedClmns:
+                for mapping in indexNameColumnMapping:
+                    if clmName == indexNameColumnMapping[1]:
+                        self.watchedIndxs.append(indexNameColumnMapping[0])
+
+        tmp = ''
+        for indx in self.watchedIndxs:
+            tmp += str(objDbRow[indx])
+
+        self.watchedHash  = str(hash(tmp))
+
+  
+

pialert/publishers/__init__.py

@@ -0,0 +1,8 @@
+""" Publishers for Pi.Alert """
+
+"""
+each publisher exposes:
+
+- check_config () returning True / False
+- send (message) returning True / Fasle 
+"""

pialert/publishers/apprise.py

@@ -0,0 +1,42 @@
+
+import json
+import subprocess
+import conf
+from helper import noti_struc
+from logger import logResult, mylog
+
+#-------------------------------------------------------------------------------
+def check_config():
+        if conf.APPRISE_URL == '' or conf.APPRISE_HOST == '':
+            mylog('none', ['[Check Config] Error: Apprise service not set up correctly. Check your pialert.conf APPRISE_* variables.'])
+            return False
+        else:
+            return True
+
+#-------------------------------------------------------------------------------
+def send (msg: noti_struc):
+    html = msg.html
+    text = msg.text
+
+    #Define Apprise compatible payload (https://github.com/caronc/apprise-api#stateless-solution)
+    payload = html
+
+    if conf.APPRISE_PAYLOAD == 'text':
+        payload = text
+
+    _json_payload={
+    "urls": conf.APPRISE_URL,
+    "title": "Pi.Alert Notifications",
+    "format": conf.APPRISE_PAYLOAD,
+    "body": payload
+    }
+
+    try:
+        # try runnning a subprocess
+        p = subprocess.Popen(["curl","-i","-X", "POST" ,"-H", "Content-Type:application/json" ,"-d", json.dumps(_json_payload), conf.APPRISE_HOST], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        stdout, stderr = p.communicate()
+        # write stdout and stderr into .log files for debugging if needed
+        logResult (stdout, stderr)      # TO-DO should be changed to mylog
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', [e.output])

pialert/publishers/email.py

@@ -0,0 +1,90 @@
+""" Pi.Alert module to send notification emails """
+
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+import smtplib
+
+
+import conf
+from helper import hide_email, noti_struc
+from logger import mylog, print_log
+
+#-------------------------------------------------------------------------------
+def check_config ():
+    if conf.SMTP_SERVER == '' or conf.REPORT_FROM == '' or conf.REPORT_TO == '':
+        mylog('none', ['[Email Check Config] Error: Email service not set up correctly. Check your pialert.conf SMTP_*, REPORT_FROM and REPORT_TO variables.'])
+        return False
+    else:
+        return True
+    
+#-------------------------------------------------------------------------------
+def send (msg: noti_struc):
+
+    pText = msg.text
+    pHTML = msg.html
+
+    mylog('debug', '[Send Email] REPORT_TO: ' + hide_email(str(conf.REPORT_TO)) + '  SMTP_USER: ' + hide_email(str(conf.SMTP_USER)))
+
+    # Compose email
+    msg = MIMEMultipart('alternative')
+    msg['Subject'] = 'Pi.Alert Report'
+    msg['From'] = conf.REPORT_FROM
+    msg['To'] = conf.REPORT_TO
+    msg.attach (MIMEText (pText, 'plain'))
+    msg.attach (MIMEText (pHTML, 'html'))
+
+    failedAt = ''
+
+    failedAt = print_log ('SMTP try')
+
+    try:
+        # Send mail
+        failedAt = print_log('Trying to open connection to ' + str(conf.SMTP_SERVER) + ':' + str(conf.SMTP_PORT))
+
+        if conf.SMTP_FORCE_SSL:
+            failedAt = print_log('SMTP_FORCE_SSL == True so using .SMTP_SSL()')
+            if conf.SMTP_PORT == 0:
+                failedAt = print_log('SMTP_PORT == 0 so sending .SMTP_SSL(SMTP_SERVER)')
+                smtp_connection = smtplib.SMTP_SSL(conf.SMTP_SERVER)
+            else:
+                failedAt = print_log('SMTP_PORT == 0 so sending .SMTP_SSL(SMTP_SERVER, SMTP_PORT)')
+                smtp_connection = smtplib.SMTP_SSL(conf.SMTP_SERVER, conf.SMTP_PORT)
+
+        else:
+            failedAt = print_log('SMTP_FORCE_SSL == False so using .SMTP()')
+            if conf.SMTP_PORT == 0:
+                failedAt = print_log('SMTP_PORT == 0 so sending .SMTP(SMTP_SERVER)')
+                smtp_connection = smtplib.SMTP (conf.SMTP_SERVER)
+            else:
+                failedAt = print_log('SMTP_PORT == 0 so sending .SMTP(SMTP_SERVER, SMTP_PORT)')
+                smtp_connection = smtplib.SMTP (conf.SMTP_SERVER, conf.SMTP_PORT)
+
+        failedAt = print_log('Setting SMTP debug level')
+
+        # Log level set to debug of the communication between SMTP server and client
+        if conf.LOG_LEVEL == 'debug':
+            smtp_connection.set_debuglevel(1)
+
+        failedAt = print_log( 'Sending .ehlo()')
+        smtp_connection.ehlo()
+
+        if not conf.SMTP_SKIP_TLS:
+            failedAt = print_log('SMTP_SKIP_TLS == False so sending .starttls()')
+            smtp_connection.starttls()
+            failedAt = print_log('SMTP_SKIP_TLS == False so sending .ehlo()')
+            smtp_connection.ehlo()
+        if not conf.SMTP_SKIP_LOGIN:
+            failedAt = print_log('SMTP_SKIP_LOGIN == False so sending .login()')
+            smtp_connection.login (conf.SMTP_USER, conf.SMTP_PASS)
+
+        failedAt = print_log('Sending .sendmail()')
+        smtp_connection.sendmail (conf.REPORT_FROM, conf.REPORT_TO, msg.as_string())
+        smtp_connection.quit()
+    except smtplib.SMTPAuthenticationError as e:
+        mylog('none', ['      ERROR: Failed at - ', failedAt])
+        mylog('none', ['      ERROR: Couldn\'t connect to the SMTP server (SMTPAuthenticationError), skipping Email (enable LOG_LEVEL=debug for more logging)'])
+    except smtplib.SMTPServerDisconnected as e:
+        mylog('none', ['      ERROR: Failed at - ', failedAt])
+        mylog('none', ['      ERROR: Couldn\'t connect to the SMTP server (SMTPServerDisconnected), skipping Email (enable LOG_LEVEL=debug for more logging)'])
+
+    mylog('debug', '[Send Email] Last executed - ' + str(failedAt))

pialert/publishers/mqtt.py

@@ -0,0 +1,245 @@
+
+import time
+import re
+from paho.mqtt import client as mqtt_client
+
+import conf
+from logger import mylog
+from database import get_all_devices, get_device_stats
+from helper import bytes_to_string, sanitize_string
+
+
+
+#-------------------------------------------------------------------------------
+# MQTT
+#-------------------------------------------------------------------------------
+
+#-------------------------------------------------------------------------------
+def check_config():
+        if conf.MQTT_BROKER == '' or conf.MQTT_PORT == '' or conf.MQTT_USER == '' or conf.MQTT_PASSWORD == '':
+            mylog('none', ['[Check Config] Error: MQTT service not set up correctly. Check your pialert.conf MQTT_* variables.'])
+            return False
+        else:
+            return True
+
+
+#-------------------------------------------------------------------------------
+class sensor_config:
+    def __init__(self, deviceId, deviceName, sensorType, sensorName, icon):
+        self.deviceId = deviceId
+        self.deviceName = deviceName
+        self.sensorType = sensorType
+        self.sensorName = sensorName
+        self.icon = icon 
+        self.hash = str(hash(str(deviceId) + str(deviceName)+ str(sensorType)+ str(sensorName)+ str(icon)))
+
+#-------------------------------------------------------------------------------
+
+def publish_mqtt(client, topic, message):
+    status = 1
+    while status != 0:
+        result = client.publish(
+                topic=topic,
+                payload=message,
+                qos=conf.MQTT_QOS,
+                retain=True,
+            )
+
+        status = result[0]
+
+        if status != 0:            
+            mylog('info', ["Waiting to reconnect to MQTT broker"])
+            time.sleep(0.1) 
+    return True
+
+#-------------------------------------------------------------------------------
+def create_generic_device(client):  
+    
+    deviceName = 'PiAlert'
+    deviceId = 'pialert'    
+    
+    create_sensor(client, deviceId, deviceName, 'sensor', 'online', 'wifi-check')    
+    create_sensor(client, deviceId, deviceName, 'sensor', 'down', 'wifi-cancel')        
+    create_sensor(client, deviceId, deviceName, 'sensor', 'all', 'wifi')
+    create_sensor(client, deviceId, deviceName, 'sensor', 'archived', 'wifi-lock')
+    create_sensor(client, deviceId, deviceName, 'sensor', 'new', 'wifi-plus')
+    create_sensor(client, deviceId, deviceName, 'sensor', 'unknown', 'wifi-alert')
+        
+
+#-------------------------------------------------------------------------------
+def create_sensor(client, deviceId, deviceName, sensorType, sensorName, icon):    
+
+    new_sensor_config = sensor_config(deviceId, deviceName, sensorType, sensorName, icon)
+
+    # check if config already in list and if not, add it, otherwise skip
+    is_unique = True
+
+    for sensor in conf.mqtt_sensors:
+        if sensor.hash == new_sensor_config.hash:
+            is_unique = False
+            break
+           
+    # save if unique
+    if is_unique:             
+        publish_sensor(client, new_sensor_config)        
+
+
+
+
+#-------------------------------------------------------------------------------
+def publish_sensor(client, sensorConf):         
+
+    message = '{ \
+                "name":"'+ sensorConf.deviceName +' '+sensorConf.sensorName+'", \
+                "state_topic":"system-sensors/'+sensorConf.sensorType+'/'+sensorConf.deviceId+'/state", \
+                "value_template":"{{value_json.'+sensorConf.sensorName+'}}", \
+                "unique_id":"'+sensorConf.deviceId+'_sensor_'+sensorConf.sensorName+'", \
+                "device": \
+                    { \
+                        "identifiers": ["'+sensorConf.deviceId+'_sensor"], \
+                        "manufacturer": "PiAlert", \
+                        "name":"'+sensorConf.deviceName+'" \
+                    }, \
+                "icon":"mdi:'+sensorConf.icon+'" \
+                }'
+
+    topic='homeassistant/'+sensorConf.sensorType+'/'+sensorConf.deviceId+'/'+sensorConf.sensorName+'/config'
+
+    # add the sensor to the global list to keep track of succesfully added sensors
+    if publish_mqtt(client, topic, message):        
+                                     # hack - delay adding to the queue in case the process is 
+        time.sleep(conf.MQTT_DELAY_SEC)   # restarted and previous publish processes aborted 
+                                     # (it takes ~2s to update a sensor config on the broker)
+        conf.mqtt_sensors.append(sensorConf)
+
+#-------------------------------------------------------------------------------
+def mqtt_create_client():
+    def on_disconnect(client, userdata, rc):
+        conf.mqtt_connected_to_broker = False
+        
+        # not sure is below line is correct / necessary        
+        # client = mqtt_create_client() 
+
+    def on_connect(client, userdata, flags, rc):
+        
+        if rc == 0: 
+            mylog('verbose', ["        Connected to broker"])            
+            conf.mqtt_connected_to_broker = True     # Signal connection 
+        else: 
+            mylog('none', ["        Connection failed"])
+            conf.mqtt_connected_to_broker = False
+
+
+    client = mqtt_client.Client('PiAlert')   # Set Connecting Client ID    
+    client.username_pw_set(conf.MQTT_USER, conf.MQTT_PASSWORD)    
+    client.on_connect = on_connect
+    client.on_disconnect = on_disconnect
+    client.connect(conf.MQTT_BROKER, conf.MQTT_PORT)
+    client.loop_start() 
+
+    return client
+
+#-------------------------------------------------------------------------------
+def mqtt_start(db):    
+
+    #global client
+
+    if conf.mqtt_connected_to_broker == False:
+        conf.mqtt_connected_to_broker = True           
+        conf.client = mqtt_create_client() 
+    
+    client = conf.client
+    # General stats    
+
+    # Create a generic device for overal stats
+    create_generic_device(client)
+
+    # Get the data
+    row = get_device_stats(db)   
+
+    columns = ["online","down","all","archived","new","unknown"]
+
+    payload = ""
+
+    # Update the values 
+    for column in columns:       
+        payload += '"'+column+'": ' + str(row[column]) +','       
+
+    # Publish (warap into {} and remove last ',' from above)
+    publish_mqtt(client, "system-sensors/sensor/pialert/state",              
+            '{ \
+                '+ payload[:-1] +'\
+            }'
+        )
+
+
+    # Specific devices
+
+    # Get all devices
+    devices = get_all_devices(db)
+
+    sec_delay = len(devices) * int(conf.MQTT_DELAY_SEC)*5
+
+    mylog('info', ["        Estimated delay: ", (sec_delay), 's ', '(', round(sec_delay/60,1) , 'min)' ])
+
+    for device in devices:        
+
+        # Create devices in Home Assistant - send config messages
+        deviceId = 'mac_' + device["dev_MAC"].replace(" ", "").replace(":", "_").lower()
+        deviceNameDisplay = re.sub('[^a-zA-Z0-9-_\s]', '', device["dev_Name"]) 
+
+        create_sensor(client, deviceId, deviceNameDisplay, 'sensor', 'last_ip', 'ip-network')
+        create_sensor(client, deviceId, deviceNameDisplay, 'binary_sensor', 'is_present', 'wifi')
+        create_sensor(client, deviceId, deviceNameDisplay, 'sensor', 'mac_address', 'folder-key-network')
+        create_sensor(client, deviceId, deviceNameDisplay, 'sensor', 'is_new', 'bell-alert-outline')
+        create_sensor(client, deviceId, deviceNameDisplay, 'sensor', 'vendor', 'cog')
+    
+        # update device sensors in home assistant              
+
+        publish_mqtt(client, 'system-sensors/sensor/'+deviceId+'/state', 
+            '{ \
+                "last_ip": "' + device["dev_LastIP"] +'", \
+                "is_new": "' + str(device["dev_NewDevice"]) +'", \
+                "vendor": "' + sanitize_string(device["dev_Vendor"]) +'", \
+                "mac_address": "' + str(device["dev_MAC"]) +'" \
+            }'
+        ) 
+
+        publish_mqtt(client, 'system-sensors/binary_sensor/'+deviceId+'/state', 
+            '{ \
+                "is_present": "' + to_binary_sensor(str(device["dev_PresentLastScan"])) +'"\
+            }'
+        ) 
+
+        # delete device / topic
+        #  homeassistant/sensor/mac_44_ef_bf_c4_b1_af/is_present/config
+        # client.publish(
+        #     topic="homeassistant/sensor/"+deviceId+"/is_present/config",
+        #     payload="",
+        #     qos=1,
+        #     retain=True,
+        # )        
+    # time.sleep(10)
+
+
+#===============================================================================
+# Home Assistant UTILs
+#===============================================================================
+def to_binary_sensor(input):
+    # In HA a binary sensor returns ON or OFF    
+    result = "OFF"
+
+    # bytestring
+    if isinstance(input, str):
+        if input == "1":
+            result = "ON"
+    elif isinstance(input, int):
+        if input == 1:
+            result = "ON"
+    elif isinstance(input, bool):
+        if input == True:
+            result = "ON"
+    elif isinstance(input, bytes):
+        if bytes_to_string(input) == "1":
+            result = "ON"
+    return result

pialert/publishers/ntfy.py

@@ -0,0 +1,43 @@
+
+import conf
+import requests
+from base64 import b64encode
+
+from logger import mylog
+from helper import noti_struc
+
+#-------------------------------------------------------------------------------
+def check_config():
+    if conf.NTFY_HOST == '' or conf.NTFY_TOPIC == '':
+        mylog('none', ['[Check Config] Error: NTFY service not set up correctly. Check your pialert.conf NTFY_* variables.'])
+        return False
+    else:
+        return True
+    
+#-------------------------------------------------------------------------------
+def send  (msg: noti_struc):
+
+    headers = {
+        "Title": "Pi.Alert Notification",
+        "Actions": "view, Open Dashboard, "+ conf.REPORT_DASHBOARD_URL,
+        "Priority": "urgent",
+        "Tags": "warning"
+    }
+    # if username and password are set generate hash and update header
+    if conf.NTFY_USER != "" and conf.NTFY_PASSWORD != "":
+	# Generate hash for basic auth
+        # usernamepassword = "{}:{}".format(conf.NTFY_USER,conf.NTFY_PASSWORD)
+        basichash = b64encode(bytes(conf.NTFY_USER + ':' + conf.NTFY_PASSWORD, "utf-8")).decode("ascii")
+
+	# add authorization header with hash
+        headers["Authorization"] = "Basic {}".format(basichash)
+
+    try:
+        requests.post("{}/{}".format( conf.NTFY_HOST, conf.NTFY_TOPIC),
+                                        data=msg.text,
+                                        headers=headers)
+    except requests.exceptions.RequestException as e:  
+        mylog('none', ['[NTFY] Error: ', e])
+        return -1
+
+    return 0    

pialert/publishers/pushsafer.py

@@ -0,0 +1,33 @@
+
+import requests
+
+
+import conf
+from helper import  noti_struc
+from logger import mylog
+
+#-------------------------------------------------------------------------------
+def check_config():
+        if conf.PUSHSAFER_TOKEN == 'ApiKey':
+            mylog('none', ['[Check Config] Error: Pushsafer service not set up correctly. Check your pialert.conf PUSHSAFER_TOKEN variable.'])
+            return False
+        else:
+            return True
+
+#-------------------------------------------------------------------------------
+def send ( msg:noti_struc ):
+    _Text = msg.text
+    url = 'https://www.pushsafer.com/api'
+    post_fields = {
+        "t" : 'Pi.Alert Message',
+        "m" : _Text,
+        "s" : 11,
+        "v" : 3,
+        "i" : 148,
+        "c" : '#ef7f7f',
+        "d" : 'a',
+        "u" : conf.REPORT_DASHBOARD_URL,
+        "ut" : 'Open Pi.Alert',
+        "k" : conf.PUSHSAFER_TOKEN,
+        }
+    requests.post(url, data=post_fields)

pialert/publishers/webhook.py

@@ -0,0 +1,98 @@
+import json
+import subprocess
+
+import conf
+from const import logPath
+from helper import noti_struc, write_file
+from logger import logResult, mylog
+
+#-------------------------------------------------------------------------------
+def check_config():
+        if conf.WEBHOOK_URL == '':
+            mylog('none', ['[Check Config] Error: Webhook service not set up correctly. Check your pialert.conf WEBHOOK_* variables.'])
+            return False
+        else:
+            return True
+        
+#-------------------------------------------------------------------------------        
+
+def send (msg: noti_struc):
+
+    # use data type based on specified payload type
+    if conf.WEBHOOK_PAYLOAD == 'json':
+        payloadData = msg.json
+    if conf.WEBHOOK_PAYLOAD == 'html':
+        payloadData = msg.html
+    if conf.WEBHOOK_PAYLOAD == 'text':
+        payloadData = to_text(msg.json)  # TO DO can we just send msg.text?
+
+    # Define slack-compatible payload
+    _json_payload = { "text": payloadData } if conf.WEBHOOK_PAYLOAD == 'text' else {
+    "username": "Pi.Alert",
+    "text": "There are new notifications",
+    "attachments": [{
+      "title": "Pi.Alert Notifications",
+      "title_link": conf.REPORT_DASHBOARD_URL,
+      "text": payloadData
+    }]
+    }
+
+    # DEBUG - Write the json payload into a log file for debugging
+    write_file (logPath + '/webhook_payload.json', json.dumps(_json_payload))
+
+    # Using the Slack-Compatible Webhook endpoint for Discord so that the same payload can be used for both
+    if(conf.WEBHOOK_URL.startswith('https://discord.com/api/webhooks/') and not conf.WEBHOOK_URL.endswith("/slack")):
+        _WEBHOOK_URL = f"{conf.WEBHOOK_URL}/slack"
+        curlParams = ["curl","-i","-H", "Content-Type:application/json" ,"-d", json.dumps(_json_payload), _WEBHOOK_URL]
+    else:
+        _WEBHOOK_URL = conf.WEBHOOK_URL
+        curlParams = ["curl","-i","-X", conf.WEBHOOK_REQUEST_METHOD ,"-H", "Content-Type:application/json" ,"-d", json.dumps(_json_payload), _WEBHOOK_URL]
+
+    # execute CURL call
+    try:
+        # try runnning a subprocess
+        mylog('debug', ['[send_webhook] curlParams: ', curlParams])
+        p = subprocess.Popen(curlParams, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+        stdout, stderr = p.communicate()
+
+        # write stdout and stderr into .log files for debugging if needed
+        logResult (stdout, stderr)     # TO-DO should be changed to mylog
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', ['[send_webhook]', e.output])
+
+
+
+
+
+#-------------------------------------------------------------------------------
+def to_text(_json):
+    payloadData = ""
+    if len(_json['internet']) > 0 and 'internet' in conf.INCLUDED_SECTIONS:
+        payloadData += "INTERNET\n"
+        for event in _json['internet']:
+            payloadData += event[3] + ' on ' + event[2] + '. ' + event[4] + '. New address:' + event[1] + '\n'
+
+    if len(_json['new_devices']) > 0 and 'new_devices' in conf.INCLUDED_SECTIONS:
+        payloadData += "NEW DEVICES:\n"
+        for event in _json['new_devices']:
+            if event[4] is None:
+                event[4] = event[11]
+            payloadData += event[1] + ' - ' + event[4] + '\n'
+
+    if len(_json['down_devices']) > 0 and 'down_devices' in conf.INCLUDED_SECTIONS:
+        write_file (logPath + '/down_devices_example.log', _json['down_devices'])
+        payloadData += 'DOWN DEVICES:\n'
+        for event in _json['down_devices']:
+            if event[4] is None:
+                event[4] = event[11]
+            payloadData += event[1] + ' - ' + event[4] + '\n'
+
+    if len(_json['events']) > 0 and 'events' in conf.INCLUDED_SECTIONS:
+        payloadData += "EVENTS:\n"
+        for event in _json['events']:
+            if event[8] != "Internet":
+                payloadData += event[8] + " on " + event[1] + " " + event[3] + " at " + event[2] + "\n"
+
+    return payloadData

pialert/reporting.py

@@ -0,0 +1,525 @@
+
+
+import datetime
+import json
+
+import socket
+
+import subprocess
+import requests
+from json2table import convert
+
+# pialert modules
+import conf
+from const import pialertPath, logPath, apiPath
+from helper import noti_struc, generate_mac_links, removeDuplicateNewLines, timeNow, hide_email,  updateState, get_file_content, write_file
+from logger import logResult, mylog, print_log
+
+
+from publishers.email import (check_config as email_check_config, 
+                              send as send_email )
+from publishers.ntfy  import (check_config as ntfy_check_config,
+                              send as send_ntfy )
+from publishers.apprise import (check_config as apprise_check_config,
+                                send as send_apprise) 
+from publishers.webhook import (check_config as webhook_check_config,
+                                send as send_webhook) 
+from publishers.pushsafer import (check_config as pushsafer_check_config,
+                                send as send_pushsafer) 
+from publishers.mqtt import (check_config as mqtt_check_config,
+                             mqtt_start )
+
+
+#===============================================================================
+# REPORTING
+#===============================================================================
+# create a json for webhook and mqtt notifications to provide further integration options
+
+
+json_final = []
+
+
+#-------------------------------------------------------------------------------
+def construct_notifications(db, sqlQuery, tableTitle, skipText = False, suppliedJsonStruct = None):
+
+    if suppliedJsonStruct is None and sqlQuery == "":
+        return noti_struc("", "", "")
+
+    table_attributes = {"style" : "border-collapse: collapse; font-size: 12px; color:#70707", "width" : "100%", "cellspacing" : 0, "cellpadding" : "3px", "bordercolor" : "#C0C0C0", "border":"1"}
+    headerProps = "width='120px' style='color:blue; font-size: 16px;' bgcolor='#909090' "
+    thProps = "width='120px' style='color:#F0F0F0' bgcolor='#909090' "
+
+    build_direction = "TOP_TO_BOTTOM"
+    text_line = '{}\t{}\n'
+
+    if suppliedJsonStruct is None:
+        json_struc = db.get_table_as_json(sqlQuery)
+    else:
+        json_struc = suppliedJsonStruct
+
+    jsn  = json_struc.json
+    html = ""
+    text = ""
+
+    if len(jsn["data"]) > 0:
+        text = tableTitle + "\n---------\n"
+
+        # Convert a JSON into an HTML table
+        html = convert(jsn, build_direction=build_direction, table_attributes=table_attributes)
+        
+        # Cleanup the generated HTML table notification
+        html = format_table(html, "data", headerProps, tableTitle).replace('<ul>','<ul style="list-style:none;padding-left:0">').replace("<td>null</td>", "<td></td>")
+
+        headers = json_struc.columnNames
+
+        # prepare text-only message
+        if skipText == False:
+
+            for device in jsn["data"]:
+                for header in headers:
+                    padding = ""
+                    if len(header) < 4:
+                        padding = "\t"
+                    text += text_line.format ( header + ': ' + padding, device[header])
+                text += '\n'
+
+        #  Format HTML table headers
+        for header in headers:
+            html = format_table(html, header, thProps)
+
+    return noti_struc(jsn, text, html)
+
+
+
+
+def send_notifications (db):
+
+    sql = db.sql  #TO-DO
+    global mail_text, mail_html, json_final, changedPorts_json_struc, partial_html, partial_txt, partial_json
+
+    deviceUrl              = conf.REPORT_DASHBOARD_URL + '/deviceDetails.php?mac='
+    plugins_report         = False
+
+    # Reporting section
+    mylog('verbose', ['[Notification] Check if something to report'])
+
+    # prepare variables for JSON construction
+    json_internet = []
+    json_new_devices = []
+    json_down_devices = []
+    json_events = []
+    json_ports = []
+    json_plugins = []
+
+    # Disable reporting on events for devices where reporting is disabled based on the MAC address
+    sql.execute ("""UPDATE Events SET eve_PendingAlertEmail = 0
+                    WHERE eve_PendingAlertEmail = 1 AND eve_EventType != 'Device Down' AND eve_MAC IN
+                        (
+                            SELECT dev_MAC FROM Devices WHERE dev_AlertEvents = 0
+						)""")
+    sql.execute ("""UPDATE Events SET eve_PendingAlertEmail = 0
+                    WHERE eve_PendingAlertEmail = 1 AND eve_EventType = 'Device Down' AND eve_MAC IN
+                        (
+                            SELECT dev_MAC FROM Devices WHERE dev_AlertDeviceDown = 0
+						)""")
+
+    # Open text Template
+    mylog('verbose', ['[Notification] Open text Template'])
+    template_file = open(pialertPath + '/back/report_template.txt', 'r')
+    mail_text = template_file.read()
+    template_file.close()
+
+    # Open html Template
+    mylog('verbose', ['[Notification] Open html Template'])
+    template_file = open(pialertPath + '/back/report_template.html', 'r')
+    if conf.newVersionAvailable :
+        template_file = open(pialertPath + '/back/report_template_new_version.html', 'r')
+
+    mail_html = template_file.read()
+    template_file.close()
+
+    # Report Header & footer
+    timeFormated = timeNow().strftime ('%Y-%m-%d %H:%M')
+    mail_text = mail_text.replace ('<REPORT_DATE>', timeFormated)
+    mail_html = mail_html.replace ('<REPORT_DATE>', timeFormated)
+
+    mail_text = mail_text.replace ('<SERVER_NAME>', socket.gethostname() )
+    mail_html = mail_html.replace ('<SERVER_NAME>', socket.gethostname() )
+
+    mylog('verbose', ['[Notification] included sections: ', conf.INCLUDED_SECTIONS ])
+
+    if 'internet' in conf.INCLUDED_SECTIONS :
+        # Compose Internet Section
+        sqlQuery = """SELECT eve_MAC as MAC,  eve_IP as IP, eve_DateTime as Datetime, eve_EventType as "Event Type", eve_AdditionalInfo as "More info" FROM Events
+                        WHERE eve_PendingAlertEmail = 1 AND eve_MAC = 'Internet'
+                        ORDER BY eve_DateTime"""
+
+        notiStruc = construct_notifications(db, sqlQuery, "Internet IP change")
+
+        # collect "internet" (IP changes) for the webhook json
+        json_internet = notiStruc.json["data"]
+
+        mail_text = mail_text.replace ('<SECTION_INTERNET>', notiStruc.text + '\n')
+        mail_html = mail_html.replace ('<INTERNET_TABLE>', notiStruc.html)
+        mylog('verbose', ['[Notification] Internet sections done.'])
+
+    if 'new_devices' in conf.INCLUDED_SECTIONS :
+        # Compose New Devices Section
+        sqlQuery = """SELECT eve_MAC as MAC, eve_DateTime as Datetime, dev_LastIP as IP, eve_EventType as "Event Type", dev_Name as "Device name", dev_Comments as Comments  FROM Events_Devices
+                        WHERE eve_PendingAlertEmail = 1
+                        AND eve_EventType = 'New Device'
+                        ORDER BY eve_DateTime"""
+
+        notiStruc = construct_notifications(db, sqlQuery, "New devices")
+
+        # collect "new_devices" for the webhook json
+        json_new_devices = notiStruc.json["data"]
+
+        mail_text = mail_text.replace ('<SECTION_NEW_DEVICES>', notiStruc.text + '\n')
+        mail_html = mail_html.replace ('<NEW_DEVICES_TABLE>', notiStruc.html)
+        mylog('verbose', ['[Notification] New Devices sections done.'])
+
+    if 'down_devices' in conf.INCLUDED_SECTIONS :
+        # Compose Devices Down Section
+        sqlQuery = """SELECT eve_MAC as MAC, eve_DateTime as Datetime, dev_LastIP as IP, eve_EventType as "Event Type", dev_Name as "Device name", dev_Comments as Comments  FROM Events_Devices
+                        WHERE eve_PendingAlertEmail = 1
+                        AND eve_EventType = 'Device Down'
+                        ORDER BY eve_DateTime"""
+
+        notiStruc = construct_notifications(db, sqlQuery, "Down devices")
+
+        # collect "new_devices" for the webhook json
+        json_down_devices = notiStruc.json["data"]
+
+        mail_text = mail_text.replace ('<SECTION_DEVICES_DOWN>', notiStruc.text + '\n')
+        mail_html = mail_html.replace ('<DOWN_DEVICES_TABLE>', notiStruc.html)
+        mylog('verbose', ['[Notification] Down Devices sections done.'])
+
+    if 'events' in conf.INCLUDED_SECTIONS :
+        # Compose Events Section
+        sqlQuery = """SELECT eve_MAC as MAC, eve_DateTime as Datetime, dev_LastIP as IP, eve_EventType as "Event Type", dev_Name as "Device name", dev_Comments as Comments  FROM Events_Devices
+                        WHERE eve_PendingAlertEmail = 1
+                        AND eve_EventType IN ('Connected','Disconnected',
+                            'IP Changed')
+                        ORDER BY eve_DateTime"""
+
+        notiStruc = construct_notifications(db, sqlQuery, "Events")
+
+        # collect "events" for the webhook json
+        json_events = notiStruc.json["data"]
+
+        mail_text = mail_text.replace ('<SECTION_EVENTS>', notiStruc.text + '\n')
+        mail_html = mail_html.replace ('<EVENTS_TABLE>', notiStruc.html)
+        mylog('verbose', ['[Notification] Events sections done.'])
+
+    if 'ports' in conf.INCLUDED_SECTIONS :
+        # collect "ports" for the webhook json
+        mylog('verbose', ['[Notification] Ports: conf.changedPorts_json_struc:', conf.changedPorts_json_struc])
+        if conf.changedPorts_json_struc is not None:
+            json_ports =  conf.changedPorts_json_struc.json["data"]
+
+        notiStruc = construct_notifications(db, "", "Ports", True, conf.changedPorts_json_struc)
+        mylog('verbose', ['[Notification] Ports: notiStruc:', notiStruc ])
+        mail_html = mail_html.replace ('<PORTS_TABLE>', notiStruc.html)
+
+        portsTxt = ""
+        if conf.changedPorts_json_struc is not None:
+            portsTxt = "Ports \n---------\n Ports changed! Check PiAlert for details!\n"
+
+        mail_text = mail_text.replace ('<PORTS_TABLE>', portsTxt )
+        mylog('verbose', ['[Notification] Ports sections done.'])    
+
+    if 'plugins' in conf.INCLUDED_SECTIONS and conf.ENABLE_PLUGINS:
+        # Compose Plugins Section
+        sqlQuery = """SELECT Plugin, Object_PrimaryId, Object_SecondaryId, DateTimeChanged, Watched_Value1, Watched_Value2, Watched_Value3, Watched_Value4, Status from Plugins_Events"""
+
+        notiStruc = construct_notifications(db, sqlQuery, "Plugins")
+
+        # collect "plugins" for the webhook json
+        json_plugins = notiStruc.json["data"]
+
+        mail_text = mail_text.replace ('<PLUGINS_TABLE>', notiStruc.text + '\n')
+        mail_html = mail_html.replace ('<PLUGINS_TABLE>', notiStruc.html)
+
+        # check if we need to report something
+        plugins_report = len(json_plugins) > 0
+        mylog('verbose', ['[Notification] Plugins sections done.'])
+
+    json_final = {
+                    "internet": json_internet,
+                    "new_devices": json_new_devices,
+                    "down_devices": json_down_devices,
+                    "events": json_events,
+                    "ports": json_ports,
+                    "plugins": json_plugins,
+                    }
+
+    mail_text = removeDuplicateNewLines(mail_text)
+
+    # Create clickable MAC links
+    mail_html = generate_mac_links (mail_html, deviceUrl)
+
+    #  Write output emails for debug
+    write_file (logPath + '/report_output.json', json.dumps(json_final))
+    write_file (logPath + '/report_output.txt', mail_text)
+    write_file (logPath + '/report_output.html', mail_html)
+
+    # Send Mail
+    if json_internet != [] or json_new_devices != [] or json_down_devices != [] or json_events != [] or json_ports != [] or conf.debug_force_notification or plugins_report:
+
+        mylog('none', ['[Notification] Changes detected, sending reports'])
+
+        msg = noti_struc(json_final, mail_text, mail_html)
+
+        mylog('info', ['[Notification] Udateing API files'])
+        send_api()
+
+        if conf.REPORT_MAIL and check_config('email'):
+            updateState(db,"Send: Email")
+            mylog('info', ['[Notification] Sending report by Email'])
+            send_email (msg )
+        else :
+            mylog('verbose', ['[Notification] Skip email'])
+        if conf.REPORT_APPRISE and check_config('apprise'):
+            updateState(db,"Send: Apprise")
+            mylog('info', ['[Notification] Sending report by Apprise'])
+            send_apprise (msg)
+        else :
+            mylog('verbose', ['[Notification] Skip Apprise'])
+        if conf.REPORT_WEBHOOK and check_config('webhook'):
+            updateState(db,"Send: Webhook")
+            mylog('info', ['[Notification] Sending report by Webhook'])
+            send_webhook (msg)
+        else :
+            mylog('verbose', ['[Notification] Skip webhook'])
+        if conf.REPORT_NTFY and check_config('ntfy'):
+            updateState(db,"Send: NTFY")
+            mylog('info', ['[Notification] Sending report by NTFY'])
+            send_ntfy (msg)
+        else :
+            mylog('verbose', ['[Notification] Skip NTFY'])
+        if conf.REPORT_PUSHSAFER and check_config('pushsafer'):
+            updateState(db,"Send: PUSHSAFER")
+            mylog('info', ['[Notification] Sending report by PUSHSAFER'])
+            send_pushsafer (msg)
+        else :
+            mylog('verbose', ['[Notification] Skip PUSHSAFER'])
+        # Update MQTT entities
+        if conf.REPORT_MQTT and check_config('mqtt'):
+            updateState(db,"Send: MQTT")
+            mylog('info', ['[Notification] Establishing MQTT thread'])
+            mqtt_start(db)
+        else :
+            mylog('verbose', ['[Notification] Skip MQTT'])
+    else :
+        mylog('verbose', ['[Notification] No changes to report'])
+
+    # Clean Pending Alert Events
+    sql.execute ("""UPDATE Devices SET dev_LastNotification = ?
+                    WHERE dev_MAC IN (SELECT eve_MAC FROM Events
+                                      WHERE eve_PendingAlertEmail = 1)
+                 """, (datetime.datetime.now(),) )
+    sql.execute ("""UPDATE Events SET eve_PendingAlertEmail = 0
+                    WHERE eve_PendingAlertEmail = 1""")
+
+    # clear plugin events
+    sql.execute ("DELETE FROM Plugins_Events")
+
+    conf.changedPorts_json_struc = None
+
+    # DEBUG - print number of rows updated
+    mylog('info', ['[Notification] Notifications changes: ', sql.rowcount])
+
+    # Commit changes
+    db.commitDB()
+
+
+#-------------------------------------------------------------------------------
+def check_config(service):
+
+    if service == 'email':
+        return email_check_config()
+    
+    #    if conf.SMTP_SERVER == '' or conf.REPORT_FROM == '' or conf.REPORT_TO == '':
+    #        mylog('none', ['[Check Config] Error: Email service not set up correctly. Check your pialert.conf SMTP_*, REPORT_FROM and REPORT_TO variables.'])
+    #        return False
+    #    else:
+    #        return True
+
+    if service == 'apprise':
+        return apprise_check_config()
+    
+    #    if conf.APPRISE_URL == '' or conf.APPRISE_HOST == '':
+    #        mylog('none', ['[Check Config] Error: Apprise service not set up correctly. Check your pialert.conf APPRISE_* variables.'])
+    #        return False
+    #    else:
+    #        return True
+
+    if service == 'webhook':
+        return webhook_check_config()
+    
+    #    if conf.WEBHOOK_URL == '':
+    #        mylog('none', ['[Check Config] Error: Webhook service not set up correctly. Check your pialert.conf WEBHOOK_* variables.'])
+    #        return False
+    #    else:
+    #        return True
+
+    if service == 'ntfy':
+        return ntfy_check_config ()
+    #
+    #    if conf.NTFY_HOST == '' or conf.NTFY_TOPIC == '':
+    #        mylog('none', ['[Check Config] Error: NTFY service not set up correctly. Check your pialert.conf NTFY_* variables.'])
+    #        return False
+    #    else:
+    #        return True
+
+    if service == 'pushsafer':
+        return pushsafer_check_config()
+
+    if service == 'mqtt':
+        return mqtt_check_config()
+
+#-------------------------------------------------------------------------------
+# Replacing table headers
+def format_table (html, thValue, props, newThValue = ''):
+
+    if newThValue == '':
+        newThValue = thValue
+
+    return html.replace("<th>"+thValue+"</th>", "<th "+props+" >"+newThValue+"</th>" )
+
+#-------------------------------------------------------------------------------
+def format_report_section (pActive, pSection, pTable, pText, pHTML):
+
+
+    # Replace section text
+    if pActive :
+        conf.mail_text = conf.mail_text.replace ('<'+ pTable +'>', pText)
+        conf.mail_html = conf.mail_html.replace ('<'+ pTable +'>', pHTML)
+
+        conf.mail_text = remove_tag (conf.mail_text, pSection)
+        conf.mail_html = remove_tag (conf.mail_html, pSection)
+    else:
+        conf.mail_text = remove_section (conf.mail_text, pSection)
+        conf.mail_html = remove_section (conf.mail_html, pSection)
+
+#-------------------------------------------------------------------------------
+def remove_section (pText, pSection):
+    # Search section into the text
+    if pText.find ('<'+ pSection +'>') >=0 \
+    and pText.find ('</'+ pSection +'>') >=0 :
+        # return text without the section
+        return pText[:pText.find ('<'+ pSection+'>')] + \
+               pText[pText.find ('</'+ pSection +'>') + len (pSection) +3:]
+    else :
+        # return all text
+        return pText
+
+#-------------------------------------------------------------------------------
+def remove_tag (pText, pTag):
+    # return text without the tag
+    return pText.replace ('<'+ pTag +'>','').replace ('</'+ pTag +'>','')
+
+
+#-------------------------------------------------------------------------------
+# Reporting
+#-------------------------------------------------------------------------------
+
+#-------------------------------------------------------------------------------
+def send_api():
+        mylog('verbose', ['[Send API] Updating notification_* files in ', apiPath])
+
+        write_file(apiPath + 'notification_text.txt'  , mail_text)
+        write_file(apiPath + 'notification_text.html'  , mail_html)
+        write_file(apiPath + 'notification_json_final.json'  , json.dumps(json_final))
+
+
+#-------------------------------------------------------------------------------
+def skip_repeated_notifications (db):
+
+    # Skip repeated notifications
+    # due strfime : Overflow --> use  "strftime / 60"
+    mylog('verbose','[Skip Repeated Notifications] Skip Repeated start')
+    db.sql.execute ("""UPDATE Events SET eve_PendingAlertEmail = 0
+                    WHERE eve_PendingAlertEmail = 1 AND eve_MAC IN
+                        (
+                        SELECT dev_MAC FROM Devices
+                        WHERE dev_LastNotification IS NOT NULL
+                          AND dev_LastNotification <>""
+                          AND (strftime("%s", dev_LastNotification)/60 +
+                                dev_SkipRepeated * 60) >
+                              (strftime('%s','now','localtime')/60 )
+                        )
+                 """ )
+    mylog('verbose','[Skip Repeated Notifications] Skip Repeated end')
+
+    db.commitDB()
+
+
+#===============================================================================
+# UTIL
+#===============================================================================
+
+#-------------------------------------------------------------------------------
+def check_and_run_event(db):
+    sql = db.sql # TO-DO
+    sql.execute(""" select * from Parameters where par_ID = "Front_Event" """)
+    rows = sql.fetchall()
+
+    event, param = ['','']
+    if len(rows) > 0 and rows[0]['par_Value'] != 'finished':
+        event = rows[0]['par_Value'].split('|')[0]
+        param = rows[0]['par_Value'].split('|')[1]
+    else:
+        return
+
+    if event == 'test':
+        handle_test(param)
+    if event == 'run':
+        handle_run(param)
+
+    # clear event execution flag
+    sql.execute ("UPDATE Parameters SET par_Value='finished' WHERE par_ID='Front_Event'")
+
+    # commit to DB
+    db.commitDB()
+
+#-------------------------------------------------------------------------------
+def handle_run(runType):
+    global last_network_scan
+
+    mylog('info', ['[', timeNow(), '] START Run: ', runType])
+
+    if runType == 'ENABLE_ARPSCAN':
+        last_network_scan = conf.time_started - datetime.timedelta(hours = 24)
+
+    mylog('info', ['[', timeNow(), '] END Run: ', runType])
+
+#-------------------------------------------------------------------------------
+def handle_test(testType):
+
+    mylog('info', ['[', timeNow(), '] START Test: ', testType])
+
+    # Open text sample
+    sample_txt = get_file_content(pialertPath + '/back/report_sample.txt')
+
+    # Open html sample
+    sample_html = get_file_content(pialertPath + '/back/report_sample.html')
+
+    # Open json sample and get only the payload part
+    sample_json_payload = json.loads(get_file_content(pialertPath + '/back/webhook_json_sample.json'))[0]["body"]["attachments"][0]["text"]
+
+    sample_msg = noti_struc(sample_json_payload, sample_txt, sample_html )
+
+    if testType == 'REPORT_MAIL':
+        send_email(sample_msg)
+    if testType == 'REPORT_WEBHOOK':
+        send_webhook (sample_msg)
+    if testType == 'REPORT_APPRISE':
+        send_apprise (sample_msg)
+    if testType == 'REPORT_NTFY':
+        send_ntfy (sample_msg)
+    if testType == 'REPORT_PUSHSAFER':
+        send_pushsafer (sample_msg)
+
+    mylog('info', ['[Test Publishers] END Test: ', testType])

pialert/scanners/arpscan.py

@@ -0,0 +1,57 @@
+import re
+import subprocess
+
+from logger import mylog
+
+#-------------------------------------------------------------------------------
+def execute_arpscan (userSubnets):
+
+    # output of possible multiple interfaces
+    arpscan_output = ""
+
+    # scan each interface
+    for interface in userSubnets :            
+        arpscan_output += execute_arpscan_on_interface (interface)    
+    
+    # Search IP + MAC + Vendor as regular expresion
+    re_ip = r'(?P<ip>((2[0-5]|1[0-9]|[0-9])?[0-9]\.){3}((2[0-5]|1[0-9]|[0-9])?[0-9]))'
+    re_mac = r'(?P<mac>([0-9a-fA-F]{2}[:-]){5}([0-9a-fA-F]{2}))'
+    re_hw = r'(?P<hw>.*)'
+    re_pattern = re.compile (re_ip + '\s+' + re_mac + '\s' + re_hw)
+
+    # Create Userdict of devices
+    devices_list = [device.groupdict()
+        for device in re.finditer (re_pattern, arpscan_output)]
+    
+    # Delete duplicate MAC
+    unique_mac = [] 
+    unique_devices = [] 
+
+    for device in devices_list :
+        if device['mac'] not in unique_mac: 
+            unique_mac.append(device['mac'])
+            unique_devices.append(device)    
+
+    # return list
+    mylog('debug', ['[ARP Scan] Completed found ', len(unique_devices) ,' devices ' ])    
+    return unique_devices
+
+#-------------------------------------------------------------------------------
+def execute_arpscan_on_interface (interface):    
+    # Prepare command arguments
+    subnets = interface.strip().split()
+    # Retry is 6 to avoid false offline devices
+    mylog('debug', ['[ARP Scan] - arpscan command: sudo arp-scan --ignoredups --retry=6 ', str(subnets)])
+    arpscan_args = ['sudo', 'arp-scan', '--ignoredups', '--retry=6'] + subnets
+
+    # Execute command
+    try:
+        # try runnning a subprocess
+        result = subprocess.check_output (arpscan_args, universal_newlines=True)
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', ['[ARP Scan]', e.output])
+        result = ""
+
+    mylog('debug', ['[ARP Scan] on Interface Completed with results: ', result])
+    return result

pialert/scanners/internet.py

@@ -0,0 +1,195 @@
+""" internet related functions to support Pi.Alert """
+
+import subprocess
+import re
+
+# pialert modules
+
+import conf
+from helper import timeNow, updateState
+from logger import append_line_to_file, mylog
+from const import logPath
+
+
+
+# need to find a better way to deal with settings !
+#global DDNS_ACTIVE, DDNS_DOMAIN, DDNS_UPDATE_URL, DDNS_USER, DDNS_PASSWORD 
+
+
+#===============================================================================
+# INTERNET IP CHANGE
+#===============================================================================
+def check_internet_IP ( db ):   
+
+    # Header
+    updateState(db,"Scan: Internet IP")
+    mylog('verbose', ['[Internet IP] Check Internet IP started'])    
+
+    # Get Internet IP
+    mylog('verbose', ['[Internet IP] - Retrieving Internet IP'])
+    internet_IP = get_internet_IP(conf.DIG_GET_IP_ARG)
+    # TESTING - Force IP
+        # internet_IP = "1.2.3.4"
+
+    # Check result = IP
+    if internet_IP == "" :
+        mylog('none', ['[Internet IP]    Error retrieving Internet IP'])
+        mylog('none', ['[Internet IP]    Exiting...'])
+        return False
+    mylog('verbose', ['[Internet IP] IP:      ', internet_IP])
+
+    # Get previous stored IP
+    mylog('verbose', ['[Internet IP]    Retrieving previous IP:'])    
+    previous_IP = get_previous_internet_IP (db)
+    mylog('verbose', ['[Internet IP]      ', previous_IP])
+
+    # Check IP Change
+    if internet_IP != previous_IP :
+        mylog('info', ['[Internet IP]    New internet IP: ', internet_IP])
+        save_new_internet_IP (db, internet_IP)
+        
+    else :
+        mylog('verbose', ['[Internet IP]    No changes to perform'])    
+
+    # Get Dynamic DNS IP
+    if conf.DDNS_ACTIVE :
+        mylog('verbose', ['[DDNS]    Retrieving Dynamic DNS IP'])
+        dns_IP = get_dynamic_DNS_IP()
+
+        # Check Dynamic DNS IP
+        if dns_IP == "" or dns_IP == "0.0.0.0" :
+            mylog('none', ['[DDNS]     Error retrieving Dynamic DNS IP'])            
+        mylog('none', ['[DDNS]    ', dns_IP])
+
+        # Check DNS Change
+        if dns_IP != internet_IP :
+            mylog('none', ['[DDNS]     Updating Dynamic DNS IP'])
+            message = set_dynamic_DNS_IP ()
+            mylog('none', ['[DDNS]        ', message])            
+        else :
+            mylog('verbose', ['[DDNS]     No changes to perform'])
+    else :
+        mylog('verbose', ['[DDNS]     Skipping Dynamic DNS update'])
+
+
+
+#-------------------------------------------------------------------------------
+def get_internet_IP (DIG_GET_IP_ARG):
+    # BUGFIX #46 - curl http://ipv4.icanhazip.com repeatedly is very slow
+    # Using 'dig'
+    dig_args = ['dig', '+short'] + DIG_GET_IP_ARG.strip().split()
+    try:
+        cmd_output = subprocess.check_output (dig_args, universal_newlines=True)
+    except subprocess.CalledProcessError as e:
+        mylog('none', [e.output])
+        cmd_output = '' # no internet
+
+    # Check result is an IP
+    IP = check_IP_format (cmd_output)
+
+    # Handle invalid response
+    if IP == '':
+        IP = '0.0.0.0'
+
+    return IP
+
+#-------------------------------------------------------------------------------
+def get_previous_internet_IP (db):
+    
+    previous_IP = '0.0.0.0'
+
+    # get previous internet IP stored in DB
+    db.sql.execute ("SELECT dev_LastIP FROM Devices WHERE dev_MAC = 'Internet' ")
+    result = db.sql.fetchone()
+
+    db.commitDB()
+
+    if  result is not None and len(result) > 0 :
+        previous_IP = result[0]
+
+    # return previous IP
+    return previous_IP
+
+
+
+#-------------------------------------------------------------------------------
+def save_new_internet_IP (db, pNewIP):
+    # Log new IP into logfile
+    append_line_to_file (logPath + '/IP_changes.log',
+        '['+str(timeNow()) +']\t'+ pNewIP +'\n')
+
+    prevIp = get_previous_internet_IP(db)     
+    # Save event
+    db.sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
+                        eve_EventType, eve_AdditionalInfo,
+                        eve_PendingAlertEmail)
+                    VALUES ('Internet', ?, ?, 'Internet IP Changed',
+                        'Previous Internet IP: '|| ?, 1) """,
+                    (pNewIP, timeNow(), prevIp) )
+
+    # Save new IP
+    db.sql.execute ("""UPDATE Devices SET dev_LastIP = ?
+                    WHERE dev_MAC = 'Internet' """,
+                    (pNewIP,) )
+
+    # commit changes    
+    db.commitDB()
+    
+#-------------------------------------------------------------------------------
+def check_IP_format (pIP):
+    # Check IP format
+    IPv4SEG  = r'(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])'
+    IPv4ADDR = r'(?:(?:' + IPv4SEG + r'\.){3,3}' + IPv4SEG + r')'
+    IP = re.search(IPv4ADDR, pIP)
+
+    # Return error if not IP
+    if IP is None :
+        return ""
+
+    # Return IP
+    return IP.group(0)
+
+
+
+#-------------------------------------------------------------------------------
+def get_dynamic_DNS_IP ():
+    # Using OpenDNS server
+        # dig_args = ['dig', '+short', DDNS_DOMAIN, '@resolver1.opendns.com']
+
+    # Using default DNS server
+    dig_args = ['dig', '+short', conf.DDNS_DOMAIN]
+
+    try:
+        # try runnning a subprocess
+        dig_output = subprocess.check_output (dig_args, universal_newlines=True)
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', ['[DDNS] ERROR - ', e.output])
+        dig_output = '' # probably no internet
+
+    # Check result is an IP
+    IP = check_IP_format (dig_output)
+
+    # Handle invalid response
+    if IP == '':
+        IP = '0.0.0.0'
+
+    return IP
+
+#-------------------------------------------------------------------------------
+def set_dynamic_DNS_IP ():
+    try:
+        # try runnning a subprocess
+        # Update Dynamic IP
+        curl_output = subprocess.check_output (['curl', '-s',
+            conf.DDNS_UPDATE_URL +
+            'username='  + conf.DDNS_USER +
+            '&password=' + conf.DDNS_PASSWORD +
+            '&hostname=' + conf.DDNS_DOMAIN],
+            universal_newlines=True)
+    except subprocess.CalledProcessError as e:
+        # An error occured, handle it
+        mylog('none', ['[DDNS] ERROR - ',e.output])
+        curl_output = ""    
+    
+    return curl_output

pialert/scanners/nmapscan.py

@@ -0,0 +1,211 @@
+
+import subprocess
+
+import conf
+from const import logPath, sql_nmap_scan_all
+from helper import json_struc, timeNow, updateState
+from logger import append_line_to_file, mylog
+#-------------------------------------------------------------------------------
+
+
+
+class nmap_entry:
+    def __init__(self, mac, time, port, state, service, name = '', extra = '', index = 0):
+        self.mac = mac
+        self.time = time
+        self.port = port
+        self.state = state
+        self.service = service
+        self.name = name
+        self.extra = extra
+        self.index = index
+        self.hash = str(mac) + str(port)+ str(state)+ str(service)
+
+
+#-------------------------------------------------------------------------------
+def performNmapScan(db, devicesToScan):
+    """
+    run nmap scan on a list of devices
+    discovers open ports and keeps track existing and new open ports
+    """
+    if len(devicesToScan) > 0:
+
+        timeoutSec = conf.NMAP_TIMEOUT
+
+        devTotal = len(devicesToScan)
+
+        updateState(db,"Scan: Nmap")
+
+        mylog('verbose', ['[NMAP Scan] Scan: Nmap for max ', str(timeoutSec), 's ('+ str(round(int(timeoutSec) / 60, 1)) +'min) per device'])  
+        mylog('verbose', ["[NMAP Scan] Estimated max delay: ", (devTotal * int(timeoutSec)), 's ', '(', round((devTotal * int(timeoutSec))/60,1) , 'min)' ])
+
+        devIndex = 0
+        for device in devicesToScan:
+            # Execute command
+            output = ""
+            # prepare arguments from user supplied ones
+            nmapArgs = ['nmap'] + conf.NMAP_ARGS.split() + [device["dev_LastIP"]]
+
+            progress = ' (' + str(devIndex+1) + '/' + str(devTotal) + ')'
+
+            try:
+                # try runnning a subprocess with a forced (timeout + 30 seconds)  in case the subprocess hangs
+                output = subprocess.check_output (nmapArgs, universal_newlines=True,  stderr=subprocess.STDOUT, timeout=(timeoutSec + 30))
+            except subprocess.CalledProcessError as e:
+                # An error occured, handle it
+                mylog('none', ["[NMAP Scan] " ,e.output])
+                mylog('none', ["[NMAP Scan] Error - Nmap Scan - check logs", progress])            
+            except subprocess.TimeoutExpired as timeErr:
+                mylog('verbose', ['[NMAP Scan] Nmap TIMEOUT - the process forcefully terminated as timeout reached for ', device["dev_LastIP"], progress]) 
+
+            if output == "": # check if the subprocess failed                    
+                mylog('info', ['[NMAP Scan] Nmap FAIL for ', device["dev_LastIP"], progress ,' check logs for details']) 
+            else: 
+                mylog('verbose', ['[NMAP Scan] Nmap SUCCESS for ', device["dev_LastIP"], progress])
+
+            devIndex += 1
+            
+            #  check the last run output        
+            newLines = output.split('\n')
+
+            # regular logging
+            for line in newLines:
+                append_line_to_file (logPath + '/pialert_nmap.log', line +'\n')                
+            
+            # collect ports / new Nmap Entries
+            newEntriesTmp = []
+
+            index = 0
+            startCollecting = False
+            duration = "" 
+            for line in newLines:            
+                if 'Starting Nmap' in line:
+                    if len(newLines) > index+1 and 'Note: Host seems down' in newLines[index+1]:
+                        break # this entry is empty
+                elif 'PORT' in line and 'STATE' in line and 'SERVICE' in line:
+                    startCollecting = True
+                elif 'PORT' in line and 'STATE' in line and 'SERVICE' in line:    
+                    startCollecting = False # end reached
+                elif startCollecting and len(line.split()) == 3:                                    
+                    newEntriesTmp.append(nmap_entry(device["dev_MAC"], timeNow(), line.split()[0], line.split()[1], line.split()[2], device["dev_Name"]))
+                elif 'Nmap done' in line:
+                    duration = line.split('scanned in ')[1]            
+            index += 1
+            mylog('verbose', ['[NMAP Scan] Ports found by NMAP: ', len(newEntriesTmp)])
+            process_discovered_ports(db, device, newEntriesTmp)
+        #end for loop
+
+
+
+def process_discovered_ports(db, device, discoveredPorts):
+    """
+    process ports discovered by nmap 
+    compare to previosu ports
+    update DB
+    raise notifications
+    """
+    sql = db.sql # TO-DO
+    # previous Nmap Entries
+    oldEntries = []
+    changedPortsTmp = []
+
+    mylog('verbose', ['[NMAP Scan] Process ports found by NMAP: ', len(discoveredPorts)])
+    
+    if len(discoveredPorts) > 0:   
+
+        #  get all current NMAP ports from the DB
+        rows = db.read(sql_nmap_scan_all)
+
+        for row in rows: 
+            # only collect entries matching the current MAC address
+            if row["MAC"] == device["dev_MAC"]:
+                oldEntries.append(nmap_entry(row["MAC"], row["Time"], row["Port"], row["State"], row["Service"], device["dev_Name"], row["Extra"], row["Index"]))
+                
+        newEntries = []
+
+        # Collect all entries that don't match the ones in the DB               
+        for discoveredPort in discoveredPorts:
+
+            found = False
+            
+            #  Check the new entry is already available in oldEntries and remove from processing if yes
+            for oldEntry in oldEntries: 
+                if discoveredPort.hash == oldEntry.hash:
+                    found = True
+
+            if not found:
+                newEntries.append(discoveredPort)
+
+
+        mylog('verbose', ['[NMAP Scan] Nmap newly discovered or changed ports: ', len(newEntries)])                
+
+        # collect new ports, find the corresponding old entry and return for notification purposes
+        # also update the DB with the new values after deleting the old ones
+        if len(newEntries) > 0:
+            
+            # params to build the SQL query
+            params = []
+            indexesToDelete = ""
+
+            # Find old entry matching the new entry hash
+            for newEntry in newEntries:                   
+
+                foundEntry = None
+
+                for oldEntry in oldEntries:
+                    if oldEntry.hash == newEntry.hash:
+                        indexesToDelete = indexesToDelete + str(oldEntry.index) + ','
+                        foundEntry = oldEntry                        
+
+                columnNames = ["Name", "MAC", "Port", "State", "Service", "Extra", "NewOrOld"  ]
+
+                # Old entry found
+                if foundEntry is not None:
+                    # Build params for sql query
+                    params.append((newEntry.mac, newEntry.time, newEntry.port, newEntry.state, newEntry.service, oldEntry.extra))
+                    # Build JSON for API and notifications
+                    changedPortsTmp.append({       
+                                            "Name"      : foundEntry.name, 
+                                            "MAC"       : newEntry.mac, 
+                                            "Port"      : newEntry.port, 
+                                            "State"     : newEntry.state, 
+                                            "Service"   : newEntry.service, 
+                                            "Extra"     : foundEntry.extra,
+                                            "NewOrOld"  : "New values"
+                                        })
+                    changedPortsTmp.append({ 
+                                            "Name"      : foundEntry.name, 
+                                            "MAC"       : foundEntry.mac, 
+                                            "Port"      : foundEntry.port, 
+                                            "State"     : foundEntry.state, 
+                                            "Service"   : foundEntry.service, 
+                                            "Extra"     : foundEntry.extra,
+                                            "NewOrOld"  : "Old values"
+                                        })                            
+                # New entry - no matching Old entry found
+                else:
+                    # Build params for sql query
+                    params.append((newEntry.mac, newEntry.time, newEntry.port, newEntry.state, newEntry.service, ''))
+                    # Build JSON for API and notifications
+                    changedPortsTmp.append({
+                                            "Name"      : "New device", 
+                                            "MAC"       : newEntry.mac, 
+                                            "Port"      : newEntry.port, 
+                                            "State"     : newEntry.state, 
+                                            "Service"   : newEntry.service, 
+                                            "Extra"     : "",
+                                            "NewOrOld"  : "New device"                                       
+                                        })
+
+            conf.changedPorts_json_struc = json_struc({ "data" : changedPortsTmp}, columnNames)
+
+            #  Delete old entries if available
+            if len(indexesToDelete) > 0:
+                sql.execute ("DELETE FROM Nmap_Scan where \"Index\" in (" + indexesToDelete[:-1] +")")
+                db.commitDB()
+
+            # Insert new values into the DB 
+            sql.executemany ("""INSERT INTO Nmap_Scan ("MAC", "Time", "Port", "State", "Service", "Extra") VALUES (?, ?, ?, ?, ?, ?)""", params) 
+            db.commitDB()
+
+

pialert/scanners/pholusscan.py

@@ -0,0 +1,200 @@
+import subprocess
+import re
+
+from const import fullPholusPath, logPath
+from helper import checkIPV4, timeNow, updateState
+from logger import mylog
+
+#-------------------------------------------------------------------------------
+
+def performPholusScan (db, timeoutSec, userSubnets):
+    sql = db.sql # TO-DO
+    # scan every interface
+    for subnet in userSubnets:
+
+        temp = subnet.split("--interface=")
+
+        if len(temp) != 2:
+            mylog('none', ["[PholusScan] Skip scan (need subnet in format '192.168.1.0/24 --inteface=eth0'), got: ", subnet])
+            return
+
+        mask = temp[0].strip()
+        interface = temp[1].strip()
+
+        # logging & updating app state        
+        updateState(db,"Scan: Pholus")        
+        mylog('none', ['[PholusScan] Scan: Pholus for ', str(timeoutSec), 's ('+ str(round(int(timeoutSec) / 60, 1)) +'min)'])  
+        mylog('verbose', ["[PholusScan] Pholus scan on [interface] ", interface, " [mask] " , mask])
+        
+        # the scan always lasts 2x as long, so the desired user time from settings needs to be halved
+        adjustedTimeout = str(round(int(timeoutSec) / 2, 0)) 
+
+        #  python3 -m trace --trace /home/pi/pialert/pholus/pholus3.py eth1 -rdns_scanning  192.168.1.0/24 -stimeout 600
+        pholus_args = ['python3', fullPholusPath, interface, "-rdns_scanning", mask, "-stimeout", adjustedTimeout]
+
+        # Execute command
+        output = ""
+
+        try:
+            # try runnning a subprocess with a forced (timeout + 30 seconds)  in case the subprocess hangs
+            output = subprocess.check_output (pholus_args, universal_newlines=True,  stderr=subprocess.STDOUT, timeout=(timeoutSec + 30))
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', ['[PholusScan]', e.output])
+            mylog('none', ["[PholusScan] Error - Pholus Scan - check logs"])            
+        except subprocess.TimeoutExpired as timeErr:
+            mylog('none', ['[PholusScan] Pholus TIMEOUT - the process forcefully terminated as timeout reached']) 
+
+        if output == "": # check if the subprocess failed                    
+            mylog('none', ['[PholusScan] Scan: Pholus FAIL - check logs']) 
+        else: 
+            mylog('verbose', ['[PholusScan] Scan: Pholus SUCCESS'])
+        
+        #  check the last run output
+        f = open(logPath + '/pialert_pholus_lastrun.log', 'r+')
+        newLines = f.read().split('\n')
+        f.close()        
+
+        # cleanup - select only lines containing a separator to filter out unnecessary data
+        newLines = list(filter(lambda x: '|' in x, newLines))        
+        
+        # build SQL query parameters to insert into the DB
+        params = []
+
+        for line in newLines:
+            columns = line.split("|")
+            if len(columns) == 4:
+                params.append(( interface + " " + mask, timeNow() , columns[0].replace(" ", ""), columns[1].replace(" ", ""), columns[2].replace(" ", ""), columns[3], ''))
+
+        if len(params) > 0:                
+            sql.executemany ("""INSERT INTO Pholus_Scan ("Info", "Time", "MAC", "IP_v4_or_v6", "Record_Type", "Value", "Extra") VALUES (?, ?, ?, ?, ?, ?, ?)""", params) 
+            db.commitDB()
+
+#-------------------------------------------------------------------------------
+def cleanResult(str):
+    # alternative str.split('.')[0]
+    str = str.replace("._airplay", "")
+    str = str.replace("._tcp", "")
+    str = str.replace(".local", "")
+    str = str.replace("._esphomelib", "")
+    str = str.replace("._googlecast", "")
+    str = str.replace(".lan", "")
+    str = str.replace(".home", "")
+    str = re.sub(r'-[a-fA-F0-9]{32}', '', str)    # removing last part of e.g. Nest-Audio-ff77ff77ff77ff77ff77ff77ff77ff77
+    # remove trailing dots
+    if str.endswith('.'):
+        str = str[:-1]
+
+    return str
+
+
+# Disclaimer - I'm interfacing with a script I didn't write (pholus3.py) so it's possible I'm missing types of answers
+# it's also possible the pholus3.py script can be adjusted to provide a better output to interface with it
+# Hit me with a PR if you know how! :)
+def resolve_device_name_pholus (pMAC, pIP, allRes):
+    
+    pholusMatchesIndexes = []
+
+    index = 0
+    for result in allRes:
+        #  limiting entries used for name resolution to the ones containing the current IP (v4 only)
+        if result["MAC"] == pMAC and result["Record_Type"] == "Answer" and result["IP_v4_or_v6"] == pIP and '._googlezone' not in result["Value"]:
+            # found entries with a matching MAC address, let's collect indexes             
+            pholusMatchesIndexes.append(index)
+
+        index += 1
+
+    # return if nothing found
+    if len(pholusMatchesIndexes) == 0:
+        return -1
+
+    # we have some entries let's try to select the most useful one
+
+    # airplay matches contain a lot of information
+    # Matches for example: 
+    # Brand Tv (50)._airplay._tcp.local. TXT Class:32769 "acl=0 deviceid=66:66:66:66:66:66 features=0x77777,0x38BCB46 rsf=0x3 fv=p20.T-FFFFFF-03.1 flags=0x204 model=XXXX manufacturer=Brand serialNumber=XXXXXXXXXXX protovers=1.1 srcvers=777.77.77 pi=FF:FF:FF:FF:FF:FF psi=00000000-0000-0000-0000-FFFFFFFFFF gid=00000000-0000-0000-0000-FFFFFFFFFF gcgl=0 pk=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and '._airplay._tcp.local. TXT Class:32769' in str(allRes[i]["Value"]) :
+            return allRes[i]["Value"].split('._airplay._tcp.local. TXT Class:32769')[0]
+    
+    # second best - contains airplay
+    # Matches for example: 
+    # _airplay._tcp.local. PTR Class:IN "Brand Tv (50)._airplay._tcp.local."
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and '_airplay._tcp.local. PTR Class:IN' in allRes[i]["Value"] and ('._googlecast') not in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split('"')[1])    
+
+    # Contains PTR Class:32769
+    # Matches for example: 
+    # 3.1.168.192.in-addr.arpa. PTR Class:32769 "MyPc.local."
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'PTR Class:32769' in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split('"')[1])
+
+    # Contains AAAA Class:IN
+    # Matches for example: 
+    # DESKTOP-SOMEID.local. AAAA Class:IN "fe80::fe80:fe80:fe80:fe80"
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'AAAA Class:IN' in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split('.local.')[0])
+
+    # Contains _googlecast._tcp.local. PTR Class:IN
+    # Matches for example: 
+    # _googlecast._tcp.local. PTR Class:IN "Nest-Audio-ff77ff77ff77ff77ff77ff77ff77ff77._googlecast._tcp.local."
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and '_googlecast._tcp.local. PTR Class:IN' in allRes[i]["Value"] and ('Google-Cast-Group') not in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split('"')[1])
+
+    # Contains A Class:32769
+    # Matches for example: 
+    # Android.local. A Class:32769 "192.168.1.6"
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and ' A Class:32769' in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split(' A Class:32769')[0])
+
+    # # Contains PTR Class:IN
+    # Matches for example: 
+    # _esphomelib._tcp.local. PTR Class:IN "ceiling-light-1._esphomelib._tcp.local."
+    for i in pholusMatchesIndexes:
+        if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'PTR Class:IN' in allRes[i]["Value"]:
+            return cleanResult(allRes[i]["Value"].split('"')[1])
+
+    return -1
+    
+#-------------------------------------------------------------------------------
+
+def resolve_device_name_dig (pMAC, pIP):
+    
+    newName = ""
+
+    try :
+        dig_args = ['dig', '+short', '-x', pIP]
+
+        # Execute command
+        try:
+            # try runnning a subprocess
+            newName = subprocess.check_output (dig_args, universal_newlines=True)
+        except subprocess.CalledProcessError as e:
+            # An error occured, handle it
+            mylog('none', ['[device_name_dig] ', e.output])            
+            # newName = "Error - check logs"
+            return -1
+
+        # Check returns
+        newName = newName.strip()
+
+        if len(newName) == 0 :
+            return -1
+            
+        # Cleanup
+        newName = cleanResult(newName)
+
+        if newName == "" or  len(newName) == 0: 
+            return -1
+
+        # Return newName
+        return newName
+
+    # not Found
+    except subprocess.CalledProcessError :
+        return -1            

pialert/scanners/pihole.py

@@ -0,0 +1,94 @@
+""" module to import db and leases from PiHole """
+
+import sqlite3
+
+import conf
+from const import piholeDB, piholeDhcpleases
+from logger import mylog
+
+#-------------------------------------------------------------------------------
+def copy_pihole_network (db):
+    """
+    attach the PiHole Database and copy the PiHole_Network table accross into the PiAlert DB
+    """
+    
+    sql = db.sql # TO-DO
+    # Open Pi-hole DB
+    mylog('debug', '[PiHole Network] - attach PiHole DB')
+
+    try:
+        sql.execute ("ATTACH DATABASE '"+ piholeDB +"' AS PH")
+    except sqlite3.Error as e:
+        mylog('none',[ '[PiHole Network] - SQL ERROR: ', e])
+    
+
+    # Copy Pi-hole Network table
+
+    try:
+        sql.execute ("DELETE FROM PiHole_Network")
+
+        # just for reporting
+        new_devices = []        
+        sql.execute ( """SELECT hwaddr, macVendor, lastQuery,
+                        (SELECT name FROM PH.network_addresses
+                         WHERE network_id = id ORDER BY lastseen DESC, ip),
+                        (SELECT ip FROM PH.network_addresses
+                         WHERE network_id = id ORDER BY lastseen DESC, ip)
+                    FROM PH.network
+                    WHERE hwaddr NOT LIKE 'ip-%'
+                      AND hwaddr <> '00:00:00:00:00:00' """)
+        new_devices = sql.fetchall()
+
+        # insert into PiAlert DB
+        sql.execute ("""INSERT INTO PiHole_Network (PH_MAC, PH_Vendor, PH_LastQuery,
+                        PH_Name, PH_IP)
+                    SELECT hwaddr, macVendor, lastQuery,
+                        (SELECT name FROM PH.network_addresses
+                         WHERE network_id = id ORDER BY lastseen DESC, ip),
+                        (SELECT ip FROM PH.network_addresses
+                         WHERE network_id = id ORDER BY lastseen DESC, ip)
+                    FROM PH.network
+                    WHERE hwaddr NOT LIKE 'ip-%'
+                      AND hwaddr <> '00:00:00:00:00:00' """)
+        sql.execute ("""UPDATE PiHole_Network SET PH_Name = '(unknown)'
+                    WHERE PH_Name IS NULL OR PH_Name = '' """)
+        # Close Pi-hole DB
+        sql.execute ("DETACH PH")
+
+    except sqlite3.Error as e:
+        mylog('none',[ '[PiHole Network] - SQL ERROR: ', e])
+    
+    db.commitDB()
+    
+    mylog('debug',[ '[PiHole Network] - completed - found ', len(new_devices), ' devices'])
+    return str(sql.rowcount) != "0"
+
+
+#-------------------------------------------------------------------------------
+
+#-------------------------------------------------------------------------------
+def read_DHCP_leases (db):
+    """
+    read the PiHole DHCP file and insert all records into the DHCP_Leases table.
+    """ 
+    mylog('debug', '[PiHole DHCP] - read DHCP_Leases file')
+    # Read DHCP Leases
+    # Bugfix #1 - dhcp.leases: lines with different number of columns (5 col)
+    data = []
+    reporting = False
+    with open(piholeDhcpleases, 'r') as f:
+        for line in f:
+            reporting = True
+            row = line.rstrip().split()
+            if len(row) == 5 :
+                data.append (row)
+
+    # Insert into PiAlert table    
+    db.sql.executemany ("""INSERT INTO DHCP_Leases (DHCP_DateTime, DHCP_MAC,
+                            DHCP_IP, DHCP_Name, DHCP_MAC2)
+                        VALUES (?, ?, ?, ?, ?)
+                     """, data)
+    db.commitDB()
+
+    mylog('debug', ['[PiHole DHCP] - completed - added ',len(data), ' devices.'])
+    return reporting

pialert/scheduler.py

@@ -0,0 +1,41 @@
+""" class to manage schedules """
+import datetime
+
+from logger import mylog, print_log
+import conf
+
+#-------------------------------------------------------------------------------
+class schedule_class:
+    def __init__(self, service, scheduleObject, last_next_schedule, was_last_schedule_used, last_run = 0):
+        self.service = service
+        self.scheduleObject = scheduleObject
+        self.last_next_schedule = last_next_schedule
+        self.last_run = last_run
+        self.was_last_schedule_used = was_last_schedule_used         
+         
+    def runScheduleCheck(self):
+
+        result = False 
+
+        # Initialize the last run time if never run before
+        if self.last_run == 0:
+            self.last_run =  (datetime.datetime.now(conf.tz) - datetime.timedelta(days=365)).replace(microsecond=0)
+
+        # get the current time with the currently specified timezone
+        nowTime = datetime.datetime.now(conf.tz).replace(microsecond=0)
+
+        # Run the schedule if the current time is past the schedule time we saved last time and 
+        #               (maybe the following check is unnecessary:)
+        # if the last run is past the last time we run a scheduled Pholus scan
+        if nowTime > self.last_next_schedule and self.last_run < self.last_next_schedule:
+            mylog('debug',f'[Scheduler] - Scheduler run for {self.service}: YES')
+            self.was_last_schedule_used = True
+            result = True
+        else:
+            mylog('debug',f'[Scheduler] - Scheduler run for {self.service}: NO')
+        
+        if self.was_last_schedule_used:
+            self.was_last_schedule_used = False
+            self.last_next_schedule = self.scheduleObject.next()            
+
+        return result

test/__init__.py

@@ -0,0 +1 @@
+""" tests for Pi.Alert """

test/test_helper.py

@@ -0,0 +1,29 @@
+import sys
+import pathlib
+
+sys.path.append(str(pathlib.Path(__file__).parent.parent.resolve()) + "/pialert/")
+
+
+import datetime
+
+from helper import timeNow, updateSubnets
+
+
+# -------------------------------------------------------------------------------
+def test_helper():
+    assert timeNow() == datetime.datetime.now().replace(microsecond=0)
+
+
+# -------------------------------------------------------------------------------
+def test_updateSubnets():
+    # test single subnet
+    subnet = "192.168.1.0/24 --interface=eth0"
+    result = updateSubnets(subnet)
+    assert type(result) is list
+    assert len(result) == 1
+
+    # test multip subnets
+    subnet = ["192.168.1.0/24 --interface=eth0", "192.168.2.0/24 --interface=eth1"]
+    result = updateSubnets(subnet)
+    assert type(result) is list
+    assert len(result) == 2