vh/NetAlertX
1
0
Fork 0
mirror of https://github.com/jokob-sk/NetAlertX.git synced 2025-12-07 09:36:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Removal of DHCP_Leases and PiHole DB tables

Browse Source
This commit is contained in:
Jokob-sk
2023-08-26 12:31:49 +10:00
parent ed0276b61e
commit b114de1fd9
26 changed files with 250 additions and 226 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docker-compose.yml
View File

@@ -21,11 +21,11 @@ services:
# DELETE START anyone trying to use this file: comment out / delete BELOW lines, they are only for development purposes
- ${APP_DATA_LOCATION}/pialert/dhcp_samples/dhcp1.leases:/mnt/dhcp1.leases
- ${APP_DATA_LOCATION}/pialert/dhcp_samples/dhcp2.leases:/mnt/dhcp2.leases
- ${APP_DATA_LOCATION}/pihole/etc-pihole/pihole-FTL.db:/etc/pihole/pihole-FTL.db
- ${DEV_LOCATION}/pialert:/home/pi/pialert/pialert
- ${DEV_LOCATION}/back/report_template.html:/home/pi/pialert/back/report_template.html
- ${DEV_LOCATION}/back/report_template_new_version.html:/home/pi/pialert/back/report_template_new_version.html
- ${DEV_LOCATION}/back/report_template.txt:/home/pi/pialert/back/report_template.txt
- ${DEV_LOCATION}/pholus:/home/pi/pialert/pholus
- ${DEV_LOCATION}/dockerfiles:/home/pi/pialert/dockerfiles
- ${APP_DATA_LOCATION}/pialert/php.ini:/etc/php/7.4/fpm/php.ini
# - ${DEV_LOCATION}/front/api:/home/pi/pialert/front/api

4
docs/DATABASE.md
View File

@@ -11,13 +11,11 @@
|----------------------|----------------------| ----------------------|
| CurrentScan | Result of the current scan | ![Screen1][screen1] |
| Devices | The main devices database that also contains the Network tree mappings. If `ScanCycle` is set to `0` device is not scanned. | ![Screen2][screen2] |
| DHCP_Leases | Used for importing devices from DHCP_Leases files. Also leveraged by some plugins. | ![Screen3][screen3] |
| Events | Used to collect connection/disconnection events. | ![Screen4][screen4] |
| Nmap_Scan | Contains results of the scheduled Nmap scan, that is also displayed in the Nmap tab on each device. | ![Screen5][screen5] |
| Online_History | Used to display the `Device presence over time` chart | ![Screen6][screen6] |
| Parameters | Used to pass values between the frontend and backend. | ![Screen7][screen7] |
| Pholus_Scan | Scan results of the Pholus python network penetration script. | ![Screen8][screen8] |
| PiHole_Network | Table to copy the devices from the PiHole database | ![Screen9][screen9] |
| Plugins_Events | For capturing events exposed by a plugin via the `last_result.log` file. If unique then saved into the `Plugins_Objects` table. Entries are deleted once processed and stored in the `Plugins_History` and/or `Plugins_Objects` tables. | ![Screen10][screen10] |
| Plugins_History | History of all entries from the `Plugins_Events` table | ![Screen11][screen11] |
| Plugins_Language_Strings | Language strings colelcted from the plugin `config.json` files used for string resolution in the frontend. | ![Screen12][screen12] |
@@ -30,13 +28,11 @@
[screen1]: /docs/img/DATABASE/CurrentScan.png
[screen2]: /docs/img/DATABASE/Devices.png
[screen3]: /docs/img/DATABASE/DHCP_Leases.png
[screen4]: /docs/img/DATABASE/Events.png
[screen5]: /docs/img/DATABASE/Nmap_Scan.png
[screen6]: /docs/img/DATABASE/Online_History.png
[screen7]: /docs/img/DATABASE/Parameters.png
[screen8]: /docs/img/DATABASE/Pholus_Scan.png
[screen9]: /docs/img/DATABASE/PiHole_Network.png
[screen10]: /docs/img/DATABASE/Plugins_Events.png
[screen11]: /docs/img/DATABASE/Plugins_History.png
[screen12]: /docs/img/DATABASE/Plugins_Language_Strings.png

33
front/plugins/README.md
View File

@@ -328,12 +328,12 @@ Plugin entries can be filtered based on values entered into filter fields. The `
PiAlert will take the results of the plugin execution and insert these results into a database table, if a plugin contains the property `"mapped_to_table"` in the `config.json` root. The mapping of the columns is defined in the `database_column_definitions` array.
This approach is used to implement the `DHCPLSS` plugin. The script parses all supplied "dhcp.leases" files, get's the results in the generic table format outlined in the "Column order and values" section above and takes individual values and inserts them into the `"DHCP_Leases"` database table in the PiAlert database. All this is achieved by:
This approach is used to implement the `DHCPLSS` plugin. The script parses all supplied "dhcp.leases" files, get's the results in the generic table format outlined in the "Column order and values" section above and takes individual values and inserts them into the `"CurrentScan"` database table in the PiAlert database. All this is achieved by:
> [!NOTE]
> If results are mapped to the `CurrentScan` table, the data is then included into the regular scan loop, so for example notification for devices are sent out.
1) Specifying the database table into which the results are inserted by defining `"mapped_to_table": "DHCP_Leases"` in the root of the `config.json` file as shown below:
1) Specifying the database table into which the results are inserted by defining `"mapped_to_table": "CurrentScan"` in the root of the `config.json` file as shown below:
```json
{
@@ -342,16 +342,16 @@ This approach is used to implement the `DHCPLSS` plugin. The script parses all s
...
"data_source": "script",
"localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
...
}
```
2) Defining the target column with the `mapped_to_column` property for individual columns in the `database_column_definitions` array of the `config.json` file. For example in the `DHCPLSS` plugin, I needed to map the value of the `Object_PrimaryID` column returned by the plugin, to the `DHCP_MAC` column in the PiAlert database `DHCP_Leases` table. Notice the `"mapped_to_column": "DHCP_MAC"` key-value pair in the sample below.
2) Defining the target column with the `mapped_to_column` property for individual columns in the `database_column_definitions` array of the `config.json` file. For example in the `DHCPLSS` plugin, I needed to map the value of the `Object_PrimaryID` column returned by the plugin, to the `cur_MAC` column in the PiAlert database `CurrentScan` table. Notice the `"mapped_to_column": "cur_MAC"` key-value pair in the sample below.
```json
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
@@ -367,6 +367,29 @@ This approach is used to implement the `DHCPLSS` plugin. The script parses all s
3) That's it. PiAlert takes care of the rest. It loops thru the objects discovered by the plugin, takes the results line, by line and inserts them into the database table specified in `"mapped_to_table"`. The columns are translated from the generic plugin columns to the target table via the `"mapped_to_column"` property in the column definitions.
> [!NOTE]
> You can create a column mapping with a default value via the `mapped_to_column_data` property.
```json
{
"column": "NameDoesntMatter",
"mapped_to_column": "cur_ScanMethod",
"mapped_to_column_data": {
"value": "DHCPLSS"
},
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "MAC address"
}]
}
```
#### params
The `params` array in the `config.json` is used to enable the user to change the parameters of the executed script. For example, the user wants to monitor a specific URL.

10
front/plugins/README_ES.md
View File

@@ -327,12 +327,12 @@ Las entradas de complementos se pueden filtrar según los valores ingresados en
PiAlert tomará los resultados de la ejecución del complemento e insertará estos resultados en una tabla de base de datos, si un complemento contiene la propiedad `"mapped_to_table"` en la raíz `config.json`. El mapeo de las columnas se define en la matriz `database_column_definitions`.
Este enfoque se utiliza para implementar el complemento "DHCPLSS". El script analiza todos los archivos "dhcp.leases" proporcionados, obtiene los resultados en el formato de tabla genérica descrito en la sección "Orden y valores de las columnas" anterior y toma valores individuales y los inserta en la tabla de base de datos `"DHCP_Leases"` en PiAlert. base de datos. Todo esto se logra mediante:
Este enfoque se utiliza para implementar el complemento "DHCPLSS". El script analiza todos los archivos "dhcp.leases" proporcionados, obtiene los resultados en el formato de tabla genérica descrito en la sección "Orden y valores de las columnas" anterior y toma valores individuales y los inserta en la tabla de base de datos `"CurrentScan"` en PiAlert. base de datos. Todo esto se logra mediante:
> [!NOTE]
> Si los resultados se asignan a la tabla "CurrentScan", los datos se incluyen en el ciclo de escaneo normal, por lo que, por ejemplo, se envían notificaciones para los dispositivos.
1) Especificar la tabla de la base de datos en la que se insertan los resultados definiendo `"mapped_to_table": "DHCP_Leases"` en la raíz del archivo `config.json` como se muestra a continuación:
1) Especificar la tabla de la base de datos en la que se insertan los resultados definiendo `"mapped_to_table": "CurrentScan"` en la raíz del archivo `config.json` como se muestra a continuación:
```json
{
@@ -341,17 +341,17 @@ Este enfoque se utiliza para implementar el complemento "DHCPLSS". El script ana
...
"data_source": "script",
"localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
...
}
```
2) Definir la columna de destino con la propiedad `mapped_to_column` para columnas individuales en la matriz `database_column_definitions` del archivo `config.json`. Por ejemplo, en el complemento `DHCPLSS`, necesitaba asignar el valor de la columna `Object_PrimaryID` devuelta por el complemento a la columna `DHCP_MAC` en la tabla `DHCP_Leases` de la base de datos PiAlert. Observe el par clave-valor `"mapped_to_column": "DHCP_MAC"` en el siguiente ejemplo.
2) Definir la columna de destino con la propiedad `mapped_to_column` para columnas individuales en la matriz `database_column_definitions` del archivo `config.json`. Por ejemplo, en el complemento `DHCPLSS`, necesitaba asignar el valor de la columna `Object_PrimaryID` devuelta por el complemento a la columna `cur_MAC` en la tabla `CurrentScan` de la base de datos PiAlert. Observe el par clave-valor `"mapped_to_column": "cur_MAC"` en el siguiente ejemplo.
```json
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",

0
front/plugins/arp_scan/README_ES.md Normal file → Executable file
View File

0
front/plugins/dhcp_leases/README_ES.md Normal file → Executable file
View File

31
front/plugins/dhcp_leases/config.json
View File

@@ -14,7 +14,7 @@
],
"show_ui": true,
"localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
"display_name" : [{
"language_code":"en_us",
"string" : "DHCP Leases (Device import)"
@@ -79,7 +79,7 @@
},
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
@@ -97,7 +97,7 @@
},
{
"column": "Object_SecondaryID",
"mapped_to_column": "DHCP_IP",
"mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "device_ip",
@@ -132,7 +132,7 @@
},
{
"column": "DateTimeChanged",
"mapped_to_column": "DHCP_DateTime",
"mapped_to_column": "cur_DateTime",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
@@ -167,7 +167,7 @@
},
{
"column": "Watched_Value2",
"mapped_to_column": "DHCP_Name",
"mapped_to_column": "cur_Name",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
@@ -251,6 +251,27 @@
"string" : "Archivo DHCP leases"
}]
},
{
"column": "ScanMethod",
"mapped_to_column": "cur_ScanMethod",
"mapped_to_column_data": {
"value": "DHCPLSS"
},
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Scan method"
},
{
"language_code":"es_es",
"string" : "Método de escaneo"
}]
},
{
"column": "Status",
"css_classes": "col-sm-1",

0
front/plugins/dhcp_servers/README_ES.md Normal file → Executable file
View File

0
front/plugins/known_template__ignore/README_ES.md Normal file → Executable file
View File

0
front/plugins/newdev_template/README_ES.md Normal file → Executable file
View File

0
front/plugins/nmap_services/README_ES.md Normal file → Executable file
View File

0
front/plugins/pholus_scan/README_ES.md Normal file → Executable file
View File

0
front/plugins/pihole_scan/README_ES.md Normal file → Executable file
View File

46
front/plugins/pihole_scan/config.json
View File

@@ -162,7 +162,7 @@
}] ,
"description":[{
"language_code":"en_us",
"string" : "Send a notification if selected values change. Use <code>CTRL + Click</code> to select/deselect. <ul> <li><code>Watched_Value1</code> is IP</li><li><code>Watched_Value2</code> is Vendor</li><li><code>Watched_Value3</code> is Interface </li><li><code>Watched_Value4</code> is N/A </li></ul>"
"string" : "Send a notification if selected values change. Use <code>CTRL + Click</code> to select/deselect. <ul> <li><code>Watched_Value1</code> is IP</li><li><code>Watched_Value2</code> is Last Query</li><li><code>Watched_Value3</code> is Name </li><li><code>Watched_Value4</code> is N/A </li></ul>"
},
{
"language_code":"es_es",
@@ -237,23 +237,33 @@
}]
},
{
"column": "Watched_Value2",
"mapped_to_column": "cur_Vendor",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Vendor"
},
{
"language_code":"es_es",
"string" : "Proveedor"
}]
} ,
"column": "Watched_Value2",
"mapped_to_column": "cur_LastQuery",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Last Query"
}]
},
{
"column": "Watched_Value3",
"mapped_to_column": "cur_Name",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Name"
}]
},
{
"column": "Extra",
"mapped_to_column": "cur_ScanMethod",

0
front/plugins/set_password/README_ES.md Normal file → Executable file
View File

0
front/plugins/snmp_discovery/README_ES.md Normal file → Executable file
View File

27
front/plugins/snmp_discovery/config.json
View File

@@ -14,7 +14,7 @@
],
"show_ui": true,
"localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
"display_name" : [{
"language_code": "en_us",
"string": "SNMP discovery"
@@ -92,7 +92,7 @@
},
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
@@ -110,7 +110,7 @@
},
{
"column": "Object_SecondaryID",
"mapped_to_column": "DHCP_IP",
"mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "device_ip",
@@ -128,6 +128,7 @@
} ,
{
"column": "DateTimeCreated",
"mapped_to_column": "cur_DateTime",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
@@ -143,27 +144,9 @@
"string" : "Creado"
}]
},
{
"column": "DateTimeChanged",
"mapped_to_column": "DHCP_DateTime",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Changed"
},
{
"language_code":"es_es",
"string" : "Cambiado"
}]
},
{
"column": "Watched_Value1",
"mapped_to_column": "DHCP_Name",
"mapped_to_column": "cur_Name",
"css_classes": "col-sm-2",
"show": true,
"type": "label",

0
front/plugins/undiscoverables/README_ES.md Normal file → Executable file
View File

14
front/plugins/undiscoverables/config.json
View File

@@ -3,7 +3,7 @@
"unique_prefix": "UNDIS",
"enabled": true,
"data_source": "script",
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
"show_ui": true,
"localized": ["display_name", "description", "icon"],
@@ -55,7 +55,7 @@
"language_code":"en_us",
"string" : "When to run"
},
{
{
"language_code":"es_es",
"string" : "Cuándo ejecuta"
}],
@@ -63,7 +63,7 @@
"language_code":"en_us",
"string" : "When enabled, ONCE is the preferred option. It runs at startup and after every save of the config here.<br> Changes will only show in the devices <b> after the next scan!</b>"
},
{
{
"language_code":"es_es",
"string" : "Cuando está habilitado, ONCE es la opción preferida. Se ejecuta al inicio y después de cada guardado de la configuración aquí.<br> ¡Los cambios solo se mostrarán en los dispositivos <b> después del próximo escaneo!</b>"
}]
@@ -205,7 +205,7 @@
[
{
"column": "Watched_Value1",
"mapped_to_column": "DHCP_Name",
"mapped_to_column": "cur_Name",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
@@ -223,7 +223,7 @@
},
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
@@ -241,7 +241,7 @@
},
{
"column": "Object_SecondaryID",
"mapped_to_column": "DHCP_IP",
"mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "device_ip",
@@ -276,7 +276,7 @@
},
{
"column": "DateTimeChanged",
"mapped_to_column": "DHCP_DateTime",
"mapped_to_column": "cur_DateTime",
"css_classes": "col-sm-2",
"show": true,
"type": "label",

0
front/plugins/unifi_import/README_ES.md Normal file → Executable file
View File

10
front/plugins/unifi_import/config.json
View File

@@ -14,7 +14,7 @@
}
],
"localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases",
"mapped_to_table": "CurrentScan",
"display_name" : [{
"language_code":"en_us",
"string" : "UniFi import"
@@ -114,7 +114,7 @@
},
{
"column": "Object_PrimaryID",
"mapped_to_column": "DHCP_MAC",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_mac",
@@ -132,7 +132,7 @@
},
{
"column": "Object_SecondaryID",
"mapped_to_column": "DHCP_IP",
"mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "device_ip",
@@ -167,7 +167,7 @@
},
{
"column": "DateTimeChanged",
"mapped_to_column": "DHCP_DateTime",
"mapped_to_column": "cur_DateTime",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
@@ -185,7 +185,7 @@
},
{
"column": "Watched_Value1",
"mapped_to_column": "DHCP_Name",
"mapped_to_column": "cur_Name",
"css_classes": "col-sm-2",
"show": true,
"type": "label",

0
front/plugins/website_monitor/README_ES.md Normal file → Executable file
View File

3
pialert/README.md
View File

@@ -42,9 +42,6 @@ different methods to scan the network for devices or to find more details about
| Module | Description |
|--------|-----------|
|```__init__.py```| an empty init file (oops missing in the repo)|
|```arpscan.py```| run an arp-scan to discover devices |
|```internet.py```| discover the internet interface and check the external IP also manage Dynamic DNS |
|```nmapscan.py```| use Nmap to discover more about devices |
|```pholusscan.py```| use a 3rd party script Pholus to detect more details about the devices on the network |
|```pihole.py```| Use the PiHole network table in its db and also read the DHCP leases file to discover new devices |

23
pialert/database.py
View File

@@ -432,10 +432,31 @@ class DB():
cur_MAC STRING(50) NOT NULL COLLATE NOCASE,
cur_IP STRING(50) NOT NULL COLLATE NOCASE,
cur_Vendor STRING(250),
cur_ScanMethod STRING(10)
cur_ScanMethod STRING(10),
cur_Name STRING(250),
cur_LastQuery STRING(250),
cur_DateTime STRING(250)
);
""")
# indicates, if DHCP_Leases table is available
DHCP_LeasesMissing = self.sql.execute("""
SELECT name FROM sqlite_master WHERE type='table'
AND name='DHCP_Leases';
""").fetchone() == None
if DHCP_LeasesMissing == False:
self.sql.execute("DROP TABLE DHCP_Leases;")
# indicates, if PiHole_Network table is available
PiHole_NetworkMissing = self.sql.execute("""
SELECT name FROM sqlite_master WHERE type='table'
AND name='PiHole_Network';
""").fetchone() == None
if PiHole_NetworkMissing == False:
self.sql.execute("DROP TABLE PiHole_Network;")
self.commitDB()

239
pialert/device.py
View File

@@ -71,63 +71,39 @@ def save_scanned_devices (db):
sql.execute (f"""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod) VALUES ( 1, '{local_mac}', '{local_ip}', Null, 'local_MAC') """)
#-------------------------------------------------------------------------------
def print_scan_stats (db):
sql = db.sql #TO-DO
# Devices Detected
sql.execute ("""SELECT COUNT(*) FROM CurrentScan""")
mylog('verbose', ['[Scan Stats] Devices Detected.......: ', str (sql.fetchone()[0]) ])
def print_scan_stats(db):
sql = db.sql # TO-DO
# Devices arp-scan
sql.execute ("""SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanMethod='arp-scan' """)
mylog('verbose', ['[Scan Stats] arp-scan detected..: ', str (sql.fetchone()[0]) ])
query = """
SELECT
(SELECT COUNT(*) FROM CurrentScan) AS devices_detected,
(SELECT COUNT(*) FROM CurrentScan WHERE NOT EXISTS (SELECT 1 FROM Devices WHERE dev_MAC = cur_MAC)) AS new_devices,
(SELECT COUNT(*) FROM Devices WHERE dev_AlertDeviceDown = 1 AND NOT EXISTS (SELECT 1 FROM CurrentScan WHERE dev_MAC = cur_MAC)) AS down_alerts,
(SELECT COUNT(*) FROM Devices WHERE dev_AlertDeviceDown = 1 AND dev_PresentLastScan = 1 AND NOT EXISTS (SELECT 1 FROM CurrentScan WHERE dev_MAC = cur_MAC)) AS new_down_alerts,
(SELECT COUNT(*) FROM Devices WHERE dev_PresentLastScan = 0) AS new_connections,
(SELECT COUNT(*) FROM Devices WHERE dev_PresentLastScan = 1 AND NOT EXISTS (SELECT 1 FROM CurrentScan WHERE dev_MAC = cur_MAC)) AS disconnections,
(SELECT COUNT(*) FROM Devices, CurrentScan WHERE dev_MAC = cur_MAC AND dev_LastIP <> cur_IP) AS ip_changes,
cur_ScanMethod,
COUNT(*) AS scan_method_count
FROM CurrentScan
GROUP BY cur_ScanMethod
"""
# Devices Pi-hole
sql.execute ("""SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanMethod='PiHole'""")
mylog('verbose', ['[Scan Stats] Pi-hole detected...: +' + str (sql.fetchone()[0]) ])
sql.execute(query)
stats = sql.fetchall()
# New Devices
sql.execute ("""SELECT COUNT(*) FROM CurrentScan
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = cur_MAC) """)
mylog('verbose', ['[Scan Stats] New Devices........: ' + str (sql.fetchone()[0]) ])
# Down Alerts
sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_AlertDeviceDown = 1
AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC
) """)
mylog('verbose', ['[Scan Stats] Down Alerts........: ' + str (sql.fetchone()[0]) ])
# New Down Alerts
sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_AlertDeviceDown = 1
AND dev_PresentLastScan = 1
AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC
) """)
mylog('verbose', ['[Scan Stats] New Down Alerts....: ' + str (sql.fetchone()[0]) ])
# New Connections
sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC
AND dev_PresentLastScan = 0""")
mylog('verbose', ['[Scan Stats] New Connections....: ' + str ( sql.fetchone()[0]) ])
# Disconnections
sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_PresentLastScan = 1
AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC
) """)
mylog('verbose', ['[Scan Stats] Disconnections.....: ' + str ( sql.fetchone()[0]) ])
# IP Changes
sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC
AND dev_LastIP <> cur_IP """)
mylog('verbose', ['[Scan Stats] IP Changes.........: ' + str ( sql.fetchone()[0]) ])
mylog('verbose', f'[Scan Stats] Devices Detected.......: {stats[0]["devices_detected"]}')
mylog('verbose', f'[Scan Stats] New Devices..........: {stats[0]["new_devices"]}')
mylog('verbose', f'[Scan Stats] Down Alerts..........: {stats[0]["down_alerts"]}')
mylog('verbose', f'[Scan Stats] New Down Alerts......: {stats[0]["new_down_alerts"]}')
mylog('verbose', f'[Scan Stats] New Connections......: {stats[0]["new_connections"]}')
mylog('verbose', f'[Scan Stats] Disconnections.......: {stats[0]["disconnections"]}')
mylog('verbose', f'[Scan Stats] IP Changes...........: {stats[0]["ip_changes"]}')
mylog('verbose', '[Scan Stats] Scan Method Statistics:')
for row in stats:
if row["cur_ScanMethod"] is not None:
mylog('verbose', f' {row["cur_ScanMethod"]}: {row["scan_method_count"]}')
#-------------------------------------------------------------------------------
@@ -135,7 +111,7 @@ def create_new_devices (db):
sql = db.sql # TO-DO
startTime = timeNowTZ()
# arpscan - Insert events for new devices
# Insert events for new devices from CurrentScan
mylog('debug','[New Devices] New devices - 1 Events')
sql.execute (f"""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
eve_EventType, eve_AdditionalInfo,
@@ -153,7 +129,7 @@ def create_new_devices (db):
WHERE NOT EXISTS (SELECT 1 FROM Sessions
WHERE ses_MAC = cur_MAC) """)
# arpscan - Create new devices
# Create new devices from CurrentScan
mylog('debug','[New Devices] 2 Create devices')
# default New Device values preparation
@@ -192,80 +168,82 @@ def create_new_devices (db):
'{get_setting_value('NEWDEV_dev_Icon')}'
"""
sqlQuery = f"""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
sqlQuery = f"""INSERT OR IGNORE INTO Devices (dev_MAC, dev_name, dev_Vendor,
dev_LastIP, dev_FirstConnection, dev_LastConnection,
{newDevColumns})
SELECT cur_MAC, '(unknown)', cur_Vendor, cur_IP, ?, ?,
SELECT cur_MAC,
CASE WHEN LENGTH(TRIM(cur_Name)) > 0 THEN cur_Name
ELSE '(unknown)' END,
cur_Vendor, cur_IP, ?, ?,
{newDevDefaults}
FROM CurrentScan
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = cur_MAC) """
FROM CurrentScan"""
# mylog('debug',f'[New Devices] 2 Create devices SQL: {sqlQuery}')
sql.execute (sqlQuery, (startTime, startTime) )
# Pi-hole - Insert events for new devices
# NOT STRICYLY NECESARY (Devices can be created through Current_Scan)
# NOT STRICYLY NECESARY (Devices can be created through CurrentScan)
# Bugfix #2 - Pi-hole devices w/o IP
mylog('debug','[New Devices] 3 Pi-hole Events')
sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
eve_EventType, eve_AdditionalInfo,
eve_PendingAlertEmail)
SELECT PH_MAC, IFNULL (PH_IP,'-'), ?, 'New Device',
'(Pi-Hole) ' || PH_Vendor, 1
FROM PiHole_Network
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = PH_MAC) """,
(startTime, ) )
# mylog('debug','[New Devices] 3 Pi-hole Events')
# sql.execute ("""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
# eve_EventType, eve_AdditionalInfo,
# eve_PendingAlertEmail)
# SELECT PH_MAC, IFNULL (PH_IP,'-'), ?, 'New Device',
# '(Pi-Hole) ' || PH_Vendor, 1
# FROM PiHole_Network
# WHERE NOT EXISTS (SELECT 1 FROM Devices
# WHERE dev_MAC = PH_MAC) """,
# (startTime, ) )
# Pi-hole - Create New Devices
# Bugfix #2 - Pi-hole devices w/o IP
mylog('debug','[New Devices] 4 Pi-hole Create devices')
# # Pi-hole - Create New Devices
# # Bugfix #2 - Pi-hole devices w/o IP
# mylog('debug','[New Devices] 4 Pi-hole Create devices')
sqlQuery = f"""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
dev_LastIP, dev_FirstConnection, dev_LastConnection,
{newDevColumns})
SELECT PH_MAC, PH_Name, PH_Vendor, IFNULL (PH_IP,'-'),
?, ?,
{newDevDefaults}
FROM PiHole_Network
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = PH_MAC) """
# sqlQuery = f"""INSERT INTO Devices (dev_MAC, dev_name, dev_Vendor,
# dev_LastIP, dev_FirstConnection, dev_LastConnection,
# {newDevColumns})
# SELECT PH_MAC, PH_Name, PH_Vendor, IFNULL (PH_IP,'-'),
# ?, ?,
# {newDevDefaults}
# FROM PiHole_Network
# WHERE NOT EXISTS (SELECT 1 FROM Devices
# WHERE dev_MAC = PH_MAC) """
# mylog('debug',f'[New Devices] 4 Create devices SQL: {sqlQuery}')
sql.execute (sqlQuery, (startTime, startTime) )
# DHCP Leases - Insert events for new devices
mylog('debug','[New Devices] 5 DHCP Leases Events')
# # DHCP Leases - Insert events for new devices
# mylog('debug','[New Devices] 5 DHCP Leases Events')
sql.execute (f"""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
eve_EventType, eve_AdditionalInfo,
eve_PendingAlertEmail)
SELECT DHCP_MAC, DHCP_IP, '{startTime}', 'New Device', '(DHCP lease)',1
FROM DHCP_Leases
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = DHCP_MAC) """)
# sql.execute (f"""INSERT INTO Events (eve_MAC, eve_IP, eve_DateTime,
# eve_EventType, eve_AdditionalInfo,
# eve_PendingAlertEmail)
# SELECT DHCP_MAC, DHCP_IP, '{startTime}', 'New Device', '(DHCP lease)',1
# FROM DHCP_Leases
# WHERE NOT EXISTS (SELECT 1 FROM Devices
# WHERE dev_MAC = DHCP_MAC) """)
# DHCP Leases - Create New Devices
mylog('debug','[New Devices] 6 DHCP Leases Create devices')
# # DHCP Leases - Create New Devices
# mylog('debug','[New Devices] 6 DHCP Leases Create devices')
sqlQuery = f"""INSERT INTO Devices (dev_MAC, dev_name, dev_LastIP,
dev_Vendor, dev_FirstConnection, dev_LastConnection,
{newDevColumns})
SELECT DISTINCT DHCP_MAC,
(SELECT DHCP_Name FROM DHCP_Leases AS D2
WHERE D2.DHCP_MAC = D1.DHCP_MAC
ORDER BY DHCP_DateTime DESC LIMIT 1),
(SELECT DHCP_IP FROM DHCP_Leases AS D2
WHERE D2.DHCP_MAC = D1.DHCP_MAC
ORDER BY DHCP_DateTime DESC LIMIT 1),
'(unknown)', ?, ?,
{newDevDefaults}
FROM DHCP_Leases AS D1
WHERE NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = DHCP_MAC) """
# sqlQuery = f"""INSERT INTO Devices (dev_MAC, dev_name, dev_LastIP,
# dev_Vendor, dev_FirstConnection, dev_LastConnection,
# {newDevColumns})
# SELECT DISTINCT DHCP_MAC,
# (SELECT DHCP_Name FROM DHCP_Leases AS D2
# WHERE D2.DHCP_MAC = D1.DHCP_MAC
# ORDER BY DHCP_DateTime DESC LIMIT 1),
# (SELECT DHCP_IP FROM DHCP_Leases AS D2
# WHERE D2.DHCP_MAC = D1.DHCP_MAC
# ORDER BY DHCP_DateTime DESC LIMIT 1),
# '(unknown)', ?, ?,
# {newDevDefaults}
# FROM DHCP_Leases AS D1
# WHERE NOT EXISTS (SELECT 1 FROM Devices
# WHERE dev_MAC = DHCP_MAC) """
# mylog('debug',f'[New Devices] 6 Create devices SQL: {sqlQuery}')
@@ -279,23 +257,24 @@ def create_new_devices (db):
def update_devices_data_from_scan (db):
sql = db.sql #TO-DO
startTime = timeNowTZ()
# Update Last Connection
mylog('debug','[Update Devices] 1 Last Connection')
sql.execute (f"""UPDATE Devices SET dev_LastConnection = '{startTime}',
mylog('debug', '[Update Devices] 1 Last Connection')
sql.execute(f"""UPDATE Devices SET dev_LastConnection = '{startTime}',
dev_PresentLastScan = 1
WHERE dev_PresentLastScan = 0
AND EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC) """)
# Clean no active devices
mylog('debug','[Update Devices] 2 Clean no active devices')
sql.execute ("""UPDATE Devices SET dev_PresentLastScan = 0
mylog('debug', '[Update Devices] 2 Clean no active devices')
sql.execute("""UPDATE Devices SET dev_PresentLastScan = 0
WHERE NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC) """)
# Update IP & Vendor
mylog('debug','[Update Devices] - 3 LastIP & Vendor')
sql.execute ("""UPDATE Devices
mylog('debug', '[Update Devices] - 3 LastIP & Vendor')
sql.execute("""UPDATE Devices
SET dev_LastIP = (SELECT cur_IP FROM CurrentScan
WHERE dev_MAC = cur_MAC),
dev_Vendor = (SELECT cur_Vendor FROM CurrentScan
@@ -304,29 +283,18 @@ def update_devices_data_from_scan (db):
WHERE EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC) """)
# Pi-hole Network - Update (unknown) Name
# Update (unknown) or (name not found) Names if available
mylog('debug','[Update Devices] - 4 Unknown Name')
sql.execute ("""UPDATE Devices
SET dev_NAME = (SELECT PH_Name FROM PiHole_Network
WHERE PH_MAC = dev_MAC)
SET dev_NAME = (SELECT cur_Name FROM CurrentScan
WHERE cur_MAC = dev_MAC)
WHERE (dev_Name in ("(unknown)", "(name not found)", "" )
OR dev_Name IS NULL)
AND EXISTS (SELECT 1 FROM PiHole_Network
WHERE PH_MAC = dev_MAC
AND PH_NAME IS NOT NULL
AND PH_NAME <> '') """)
# DHCP Leases - Update (unknown) Name
sql.execute ("""UPDATE Devices
SET dev_NAME = (SELECT DHCP_Name FROM DHCP_Leases
WHERE DHCP_MAC = dev_MAC)
WHERE (dev_Name in ("(unknown)", "(name not found)", "" )
OR dev_Name IS NULL)
AND EXISTS (SELECT 1 FROM DHCP_Leases
WHERE DHCP_MAC = dev_MAC)""")
# DHCP Leases - Vendor
mylog('debug','[Update Devices] - 5 Vendor')
AND EXISTS (SELECT 1 FROM CurrentScan
WHERE cur_MAC = dev_MAC
AND cur_Name IS NOT NULL
AND cur_Name IS NOT 'null'
AND cur_Name <> '') """)
recordsToUpdate = []
query = """SELECT * FROM Devices
@@ -341,8 +309,7 @@ def update_devices_data_from_scan (db):
sql.executemany ("UPDATE Devices SET dev_Vendor = ? WHERE dev_MAC = ? ",
recordsToUpdate )
# clean-up device leases table
sql.execute ("DELETE FROM DHCP_Leases")
mylog('debug','[Update Devices] Update devices end')
#-------------------------------------------------------------------------------

14
pialert/plugin.py
View File

@@ -259,9 +259,9 @@ def execute_plugin(db, plugin, pluginsState = plugins_state() ):
# Create a tuple containing values to be inserted into the database.
# Each value corresponds to a column in the table in the order of the columns.
# must match the Plugins_Objects and Plugins_Events databse tables and can be used as input for the plugin_object_class
sqlParams.append(
sqlParams.append((
0, # "Index" placeholder
(plugin["unique_prefix"], # "Plugin"
plugin["unique_prefix"], # "Plugin"
row[0], # "Object_PrimaryID"
handle_empty(row[1]), # "Object_SecondaryID"
'null', # "DateTimeCreated" column (null placeholder)
@@ -537,7 +537,7 @@ def process_plugin_events(db, plugin, pluginsState, plugEventsArr):
raise e
# Perform database table mapping if enabled for the plugin
if len(pluginEvents) > 0 and "mapped_to_table" in plugin:
if len(pluginEvents) > 0 and "mapped_to_table" in plugin:
# Initialize an empty list to store SQL parameters.
sqlParams = []
@@ -558,6 +558,7 @@ def process_plugin_events(db, plugin, pluginsState, plugEventsArr):
for clmn in plugin['database_column_definitions']:
if 'mapped_to_column' in clmn:
mappedCols.append(clmn)
columnsStr = f'{columnsStr}, "{clmn["mapped_to_column"]}"'
valuesStr = f'{valuesStr}, ?'
@@ -598,6 +599,10 @@ def process_plugin_events(db, plugin, pluginsState, plugEventsArr):
elif col['column'] == 'Status':
tmpList.append(plgEv.status)
# Check if there's a default value specified for this column in the JSON.
if 'mapped_to_column_data' in col and 'value' in col['mapped_to_column_data']:
tmpList.append(col['mapped_to_column_data']['value'])
# Append the mapped values to the list 'sqlParams' as a tuple.
sqlParams.append(tuple(tmpList))
@@ -606,6 +611,7 @@ def process_plugin_events(db, plugin, pluginsState, plugEventsArr):
# Log a debug message showing the generated SQL query for mapping.
mylog('debug', ['[Plugins] SQL query for mapping: ', q])
mylog('debug', ['[Plugins] SQL sqlParams for mapping: ', sqlParams])
# Execute the SQL query using 'sql.executemany()' and the 'sqlParams' list of tuples.
# This will insert multiple rows into the database in one go.
@@ -617,9 +623,9 @@ def process_plugin_events(db, plugin, pluginsState, plugEventsArr):
if dbTable == 'CurrentScan':
pluginsState.processScan = True
db.commitDB()
return pluginsState