vh/NetAlertX
1
0
Fork 0
mirror of https://github.com/jokob-sk/NetAlertX.git synced 2025-12-07 09:36:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

ARPSCAN to plugin rewrite

Browse Source
This commit is contained in:
Jokob-sk
2023-08-06 10:50:03 +10:00
parent ef64014100
commit c2da5c56b8
15 changed files with 303 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docker-compose.yml
View File

@@ -9,7 +9,7 @@ services:
- type=registry,ref=docker.io/jokob-sk/pi.alert:buildcache - type=registry,ref=docker.io/jokob-sk/pi.alert:buildcache
container_name: pialert container_name: pialert
network_mode: host network_mode: host
restart: unless-stopped # restart: unless-stopped
volumes: volumes:
- ${APP_DATA_LOCATION}/pialert_dev/config:/home/pi/pialert/config - ${APP_DATA_LOCATION}/pialert_dev/config:/home/pi/pialert/config
# - ${APP_DATA_LOCATION}/pialert/config:/home/pi/pialert/config # - ${APP_DATA_LOCATION}/pialert/config:/home/pi/pialert/config
@@ -46,6 +46,7 @@ services:
- ${DEV_LOCATION}/front/network.php:/home/pi/pialert/front/network.php - ${DEV_LOCATION}/front/network.php:/home/pi/pialert/front/network.php
- ${DEV_LOCATION}/front/presence.php:/home/pi/pialert/front/presence.php - ${DEV_LOCATION}/front/presence.php:/home/pi/pialert/front/presence.php
- ${DEV_LOCATION}/front/settings.php:/home/pi/pialert/front/settings.php - ${DEV_LOCATION}/front/settings.php:/home/pi/pialert/front/settings.php
- ${DEV_LOCATION}/front/systeminfo.php:/home/pi/pialert/front/systeminfo.php
- ${DEV_LOCATION}/front/flows.php:/home/pi/pialert/front/flows.php - ${DEV_LOCATION}/front/flows.php:/home/pi/pialert/front/flows.php
- ${DEV_LOCATION}/front/plugins:/home/pi/pialert/front/plugins - ${DEV_LOCATION}/front/plugins:/home/pi/pialert/front/plugins
# DELETE END anyone trying to use this file: comment out / delete ABOVE lines, they are only for development purposes # DELETE END anyone trying to use this file: comment out / delete ABOVE lines, they are only for development purposes

8
front/plugins/README.md
View File

@@ -97,7 +97,7 @@ More on specifics below.
Currently, only 3 data sources are supported (valid `data_source` value). Currently, only 3 data sources are supported (valid `data_source` value).
- Script (`python-script`) - Script (`script`)
- SQL query on the PiAlert database (`pialert-db-query`) - SQL query on the PiAlert database (`pialert-db-query`)
- Template (`template`) - Template (`template`)
@@ -107,9 +107,9 @@ Currently, only 3 data sources are supported (valid `data_source` value).
>``` >```
Any of the above data sources have to return a "table" of the exact structure as outlined above. Any of the above data sources have to return a "table" of the exact structure as outlined above.
### "data_source": "python-script" ### "data_source": "script"
If the `data_source` is set to `python-script` the `CMD` setting (that you specify in the `settings` array section in the `config.json`) needs to contain an executable Linux command, that generates a `last_result.log` file. This file needs to be stored in the same folder as the plugin. If the `data_source` is set to `script` the `CMD` setting (that you specify in the `settings` array section in the `config.json`) needs to contain an executable Linux command, that generates a `last_result.log` file. This file needs to be stored in the same folder as the plugin.
The content of the `last_result.log` file needs to contain the columns as defined in the "Column order and values" section above. The order of columns can't be changed. After every scan it should contain only the results from the latest scan/execution. The content of the `last_result.log` file needs to contain the columns as defined in the "Column order and values" section above. The order of columns can't be changed. After every scan it should contain only the results from the latest scan/execution.
@@ -264,7 +264,7 @@ This approach is used to implement the `DHCPLSS` plugin. The script parses all s
"code_name": "dhcp_leases", "code_name": "dhcp_leases",
"unique_prefix": "DHCPLSS", "unique_prefix": "DHCPLSS",
... ...
"data_source": "python-script", "data_source": "script",
"localized": ["display_name", "description", "icon"], "localized": ["display_name", "description", "icon"],
"mapped_to_table": "DHCP_Leases", "mapped_to_table": "DHCP_Leases",
... ...

149
front/plugins/arp_scan/config.json
View File

@@ -2,34 +2,34 @@
"code_name": "arpscan", "code_name": "arpscan",
"unique_prefix": "ARPSCAN", "unique_prefix": "ARPSCAN",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"mapped_to_table": "DHCP_Leases", "mapped_to_table": "CurrentScan",
"localized": ["display_name", "description", "icon"], "localized": ["display_name", "description", "icon"],
"display_name": [ "display_name": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "Un-Discoverable Devices" "string": "Network scan (Arp-Scan)"
} }
], ],
"icon": [ "icon": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "<i class=\"fa-solid fa-binoculars\"></i>" "string": "<i class=\"fa-solid fa-search\"></i>"
} }
], ],
"description": [ "description": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "This plugin is to import undiscoverable devices from a file." "string": "This plugin is to execute an arp-scan on the local network"
} }
], ],
"params" : [ "params" : [
{ {
"name" : "devices", "name" : "subnets",
"type" : "setting", "type" : "setting",
"value" : "UNDIS_devices_to_import" "value" : "SCAN_SUBNETS"
}], }],
"settings": [ "settings": [
@@ -37,7 +37,7 @@
"function": "RUN", "function": "RUN",
"type": "text.select", "type": "text.select",
"default_value":"disabled", "default_value":"disabled",
"options": ["disabled", "once", "always_after_scan"], "options": ["disabled", "once", "schedule", "scan_cycle", "always_after_scan", "on_new_device"],
"localized": ["name", "description"], "localized": ["name", "description"],
"name" :[{ "name" :[{
"language_code":"en_us", "language_code":"en_us",
@@ -50,8 +50,8 @@
}, },
{ {
"function": "CMD", "function": "CMD",
"type": "text", "type": "readonly",
"default_value": "python3 /home/pi/pialert/front/plugins/undiscoverables/script.py devices={devices}", "default_value": "python3 /home/pi/pialert/front/plugins/arp_scan/script.py userSubnets={subnets}",
"options": [], "options": [],
"localized": ["name", "description"], "localized": ["name", "description"],
"name": [ "name": [
@@ -63,7 +63,7 @@
"description": [ "description": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "Command to run. This can not be changed" "string": "Command to run. This should not be changed"
} }
] ]
}, },
@@ -71,7 +71,7 @@
{ {
"function": "RUN_TIMEOUT", "function": "RUN_TIMEOUT",
"type": "integer", "type": "integer",
"default_value": 10, "default_value": 300,
"options": [], "options": [],
"localized": ["name", "description"], "localized": ["name", "description"],
"name": [ "name": [
@@ -88,28 +88,39 @@
] ]
}, },
{ {
"function": "WATCH", "function": "RUN_SCHD",
"type": "readonly", "type": "text",
"default_value": [], "default_value":"*/3 * * * *",
"options": [], "options": [],
"localized": ["name", "description"], "localized": ["name", "description"],
"name": [ "name" : [{
{ "language_code":"en_us",
"language_code": "en_us", "string" : "Schedule"
"string": "Watched" }],
} "description": [{
], "language_code":"en_us",
"description": [ "string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#ARPSCAN_RUN\"><code>ARPSCAN_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>*/3 * * * *</code> will run the scan every 3 minutes. Will be run NEXT time the time passes."
{ }]
"language_code": "en_us", },
"string": "Undiscoverable Devices can not change their status, no watch is enabled." {
} "function": "WATCH",
] "type": "text.multiselect",
"default_value":["Watched_Value1", "Watched_Value2"],
"options": ["Watched_Value1","Watched_Value2","Watched_Value3","Watched_Value4"],
"localized": ["name", "description"],
"name" :[{
"language_code":"en_us",
"string" : "Watched"
}] ,
"description":[{
"language_code":"en_us",
"string" : "Send a notification if selected values change. Use <code>CTRL + Click</code> to select/deselect. <ul> <li><code>Watched_Value1</code> is IP</li><li><code>Watched_Value2</code> is Vendor</li><li><code>Watched_Value3</code> is Interface </li><li><code>Watched_Value4</code> is N/A </li></ul>"
}]
}, },
{ {
"function": "REPORT_ON", "function": "REPORT_ON",
"type": "readonly", "type": "text.multiselect",
"default_value": [], "default_value": ["new", "watched-changed"],
"options": ["new", "watched-changed", "watched-not-changed"], "options": ["new", "watched-changed", "watched-not-changed"],
"localized": ["name", "description"], "localized": ["name", "description"],
"name": [ "name": [
@@ -121,60 +132,31 @@
"description": [ "description": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "No notifications will be sent." "string": "When should notification be sent out."
} }
] ]
}, }
{
"function": "devices_to_import",
"type": "list",
"default_value":["dummy_router"],
"options": [],
"localized": ["name", "description"],
"name" : [{
"language_code":"en_us",
"string" : "UnDiscoverable Devices"
}],
"description": [{
"language_code":"en_us",
"string" : "Devices to be added to the devices list."
}]
}
], ],
"database_column_definitions": "database_column_definitions":
[ [
{ {
"column": "Watched_Value1", "column": "Object_PrimaryID",
"mapped_to_column": "DHCP_Name", "mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2", "css_classes": "col-sm-2",
"show": true, "show": true,
"type": "label", "type": "devicemac",
"default_value":"", "default_value":"",
"options": [], "options": [],
"localized": ["name"], "localized": ["name"],
"name":[{ "name":[{
"language_code":"en_us", "language_code":"en_us",
"string" : "Device Name" "string" : "MAC"
}] }]
}, },
{ {
"column": "Object_PrimaryID", "column": "Watched_Value1",
"mapped_to_column": "DHCP_MAC", "mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "devicemac",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "MAC address"
}]
},
{
"column": "Object_SecondaryID",
"mapped_to_column": "DHCP_IP",
"css_classes": "col-sm-2", "css_classes": "col-sm-2",
"show": true, "show": true,
"type": "deviceip", "type": "deviceip",
@@ -185,6 +167,34 @@
"language_code":"en_us", "language_code":"en_us",
"string" : "IP" "string" : "IP"
}] }]
},
{
"column": "Watched_Value2",
"mapped_to_column": "cur_Vendor",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Vendor"
}]
} ,
{
"column": "Extra",
"mapped_to_column": "cur_ScanMethod",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Scan method"
}]
} , } ,
{ {
"column": "DateTimeCreated", "column": "DateTimeCreated",
@@ -200,8 +210,7 @@
}] }]
}, },
{ {
"column": "DateTimeChanged", "column": "DateTimeChanged",
"mapped_to_column": "DHCP_DateTime",
"css_classes": "col-sm-2", "css_classes": "col-sm-2",
"show": true, "show": true,
"type": "label", "type": "label",

117
front/plugins/arp_scan/script.py
View File

@@ -1,48 +1,121 @@
#!/usr/bin/env python #!/usr/bin/env python
# test script by running python script.py devices=test,dummy
import os import os
import pathlib import pathlib
import argparse import argparse
import sys import sys
import re
import subprocess
from time import strftime
sys.path.append("/home/pi/pialert/front/plugins") sys.path.append("/home/pi/pialert/front/plugins")
from plugin_helper import Plugin_Objects from plugin_helper import Plugin_Object, Plugin_Objects
CUR_PATH = str(pathlib.Path(__file__).parent.resolve()) CUR_PATH = str(pathlib.Path(__file__).parent.resolve())
LOG_FILE = os.path.join(CUR_PATH , 'script.log') LOG_FILE = os.path.join(CUR_PATH, 'script.log')
RESULT_FILE = os.path.join(CUR_PATH , 'last_result.log') RESULT_FILE = os.path.join(CUR_PATH, 'last_result.log')
def main(): def main():
# the script expects a parameter in the format of userSubnets=subnet1,subnet2,...
# the script expects a parameter in the format of devices=device1,device2,...
parser = argparse.ArgumentParser(description='Import devices from settings') parser = argparse.ArgumentParser(description='Import devices from settings')
parser.add_argument('devices', action="store", help="list of device names separated by ','") parser.add_argument('userSubnets', nargs='+', help="list of subnets with options")
values = parser.parse_args() values = parser.parse_args()
UNDIS_devices = Plugin_Objects( RESULT_FILE ) devices = Plugin_Objects(RESULT_FILE)
if values.devices: subnets_list = []
for fake_dev in values.devices.split('=')[1].split(','):
UNDIS_devices.add_object(
primaryId=fake_dev, # MAC (Device Name)
secondaryId="0.0.0.0", # IP Address (always 0.0.0.0)
watched1=fake_dev, # Device Name
watched2="",
watched3="",
watched4="",
extra="",
foreignKey="")
UNDIS_devices.write_result_file() if isinstance(values.userSubnets, list):
subnets_list = values.userSubnets
else:
subnets_list = [values.userSubnets]
unique_devices = execute_arpscan(subnets_list)
for device in unique_devices:
devices.add_object(
primaryId=device['mac'], # MAC (Device Name)
secondaryId=device['ip'], # IP Address
watched1=device['ip'], # Device Name
watched2=device.get('hw', ''), # Vendor (assuming it's in the 'hw' field)
watched3=device.get('interface', ''), # Add the interface
watched4='',
extra='arp-scan',
foreignKey="")
devices.write_result_file()
return 0 return 0
def execute_arpscan(userSubnets):
# output of possible multiple interfaces
arpscan_output = ""
devices_list = []
# scan each interface
for interface in userSubnets :
arpscan_output = execute_arpscan_on_interface (interface)
print(arpscan_output)
# Search IP + MAC + Vendor as regular expresion
re_ip = r'(?P<ip>((2[0-5]|1[0-9]|[0-9])?[0-9]\.){3}((2[0-5]|1[0-9]|[0-9])?[0-9]))'
re_mac = r'(?P<mac>([0-9a-fA-F]{2}[:-]){5}([0-9a-fA-F]{2}))'
re_hw = r'(?P<hw>.*)'
re_pattern = re.compile (re_ip + '\s+' + re_mac + '\s' + re_hw)
devices_list_tmp = [
{**device.groupdict(), "interface": interface}
for device in re.finditer(re_pattern, arpscan_output)
]
devices_list += devices_list_tmp
# mylog('debug', ['[ARP Scan] Found: Devices including duplicates ', len(devices_list) ])
# Delete duplicate MAC
unique_mac = []
unique_devices = []
for device in devices_list :
if device['mac'] not in unique_mac:
unique_mac.append(device['mac'])
unique_devices.append(device)
# return list
# mylog('debug', ['[ARP Scan] Found: Devices without duplicates ', len(unique_devices) ])
print("Devices List len:", len(devices_list)) # Add this line to print devices_list
print("Devices List:", devices_list) # Add this line to print devices_list
return devices_list
def execute_arpscan_on_interface(interface):
# Prepare command arguments
arpscan_args = ['sudo', 'arp-scan', '--ignoredups', '--retry=6'] + interface.split()
# Execute command
try:
# try running a subprocess safely
result = subprocess.check_output(arpscan_args, universal_newlines=True)
except subprocess.CalledProcessError as e:
# An error occurred, handle it
error_type = type(e).__name__ # Capture the error type
result = ""
return result
#=============================================================================== #===============================================================================
# BEGIN # BEGIN
#=============================================================================== #===============================================================================
if __name__ == '__main__': if __name__ == '__main__':
main() main()

2
front/plugins/dhcp_leases/config.json
View File

@@ -2,7 +2,7 @@
"code_name": "dhcp_leases", "code_name": "dhcp_leases",
"unique_prefix": "DHCPLSS", "unique_prefix": "DHCPLSS",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"data_filters": [ "data_filters": [
{ {
"compare_column" : "Object_PrimaryID", "compare_column" : "Object_PrimaryID",

2
front/plugins/dhcp_servers/config.json
View File

@@ -2,7 +2,7 @@
"code_name": "dhcp_servers", "code_name": "dhcp_servers",
"unique_prefix": "DHCPSRVS", "unique_prefix": "DHCPSRVS",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"localized": ["display_name", "description", "icon"], "localized": ["display_name", "description", "icon"],
"display_name" : [{ "display_name" : [{
"language_code":"en_us", "language_code":"en_us",

9
front/plugins/known_template/config.json
View File

@@ -48,9 +48,8 @@
], ],
"settings":[ "settings":[
{ {
"function": "flows", "function": "FLOW",
"type": "json", "type": "json",
"maxLength": 50,
"default_value": [{ "default_value": [{
"name":"apply_template", "name":"apply_template",
"trigger": [ "trigger": [
@@ -121,13 +120,13 @@
"name": [ "name": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "Flows" "string": "Plugin flow"
} }
], ],
"description": [ "description": [
{ {
"language_code": "en_us", "language_code": "en_us",
"string": "The flow." "string": "This flow makes sure the template is applied to devices that are older than 3 days."
} }
] ]
}, },

2
front/plugins/undiscoverables/config.json
View File

@@ -2,7 +2,7 @@
"code_name": "undiscoverables", "code_name": "undiscoverables",
"unique_prefix": "UNDIS", "unique_prefix": "UNDIS",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"mapped_to_table": "DHCP_Leases", "mapped_to_table": "DHCP_Leases",
"localized": ["display_name", "description", "icon"], "localized": ["display_name", "description", "icon"],

2
front/plugins/unifi_import/config.json
View File

@@ -2,7 +2,7 @@
"code_name": "unifi_import", "code_name": "unifi_import",
"unique_prefix": "UNFIMP", "unique_prefix": "UNFIMP",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"data_filters": [ "data_filters": [
{ {
"compare_column" : "Object_PrimaryID", "compare_column" : "Object_PrimaryID",

2
front/plugins/website_monitor/config.json
View File

@@ -2,7 +2,7 @@
"code_name": "website_monitor", "code_name": "website_monitor",
"unique_prefix": "WEBMON", "unique_prefix": "WEBMON",
"enabled": true, "enabled": true,
"data_source": "python-script", "data_source": "script",
"localized": ["display_name", "description", "icon"], "localized": ["display_name", "description", "icon"],
"display_name" : [{ "display_name" : [{
"language_code":"en_us", "language_code":"en_us",

3
pialert/__main__.py
View File

@@ -213,6 +213,7 @@ def main ():
nmapSchedule.last_run = timeNow() nmapSchedule.last_run = timeNow()
performNmapScan(db, get_all_devices(db)) performNmapScan(db, get_all_devices(db))
# todo replace the scans with plugins
# Perform a network scan via arp-scan or pihole # Perform a network scan via arp-scan or pihole
if last_network_scan + datetime.timedelta(minutes=conf.SCAN_CYCLE_MINUTES) < loop_start_time: if last_network_scan + datetime.timedelta(minutes=conf.SCAN_CYCLE_MINUTES) < loop_start_time:
last_network_scan = loop_start_time last_network_scan = loop_start_time
@@ -250,7 +251,7 @@ def main ():
# -------------------------------------------------- # --------------------------------------------------
# process all the scanned data into new devices # process all the scanned data into new devices
mylog('debug', "[MAIN] start processig scan results") mylog('debug', "[MAIN] start processig scan results")
process_scan (db, conf.arpscan_devices ) process_scan (db)
# Reporting # Reporting
if conf.cycle in conf.check_report: if conf.cycle in conf.check_report:

146
pialert/device.py
View File

@@ -14,44 +14,48 @@ from scanners.pholusscan import performPholusScan, resolve_device_name_dig, reso
#------------------------------------------------------------------------------- #-------------------------------------------------------------------------------
def save_scanned_devices (db, p_arpscan_devices, p_cycle_interval): def save_scanned_devices (db):
sql = db.sql #TO-DO sql = db.sql #TO-DO
cycle = 1 # always 1, only one cycle supported cycle = 1 # always 1, only one cycle supported
mylog('debug', ['[ARP Scan] Detected devices:', len(p_arpscan_devices)]) # mylog('debug', ['[ARP Scan] Detected devices:', len(p_arpscan_devices)])
# Delete previous scan data # handled by the ARPSCAN plugin
sql.execute ("DELETE FROM CurrentScan WHERE cur_ScanCycle = ?", # # Delete previous scan data
(cycle,)) # sql.execute ("DELETE FROM CurrentScan")
if len(p_arpscan_devices) > 0: # if len(p_arpscan_devices) > 0:
# Insert new arp-scan devices # # Insert new arp-scan devices
sql.executemany ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, "+ # sql.executemany ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, "+
" cur_IP, cur_Vendor, cur_ScanMethod) "+ # " cur_IP, cur_Vendor, cur_ScanMethod) "+
"VALUES ("+ str(cycle) + ", :mac, :ip, :hw, 'arp-scan')", # "VALUES (1, :mac, :ip, :hw, 'arp-scan')",
p_arpscan_devices) # p_arpscan_devices)
# ------------------------ TO CONVERT INTO PLUGIN
# # Insert Pi-hole devices
# startTime = timeNow()
# sql.execute ("""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC,
# cur_IP, cur_Vendor, cur_ScanMethod)
# SELECT ?, PH_MAC, PH_IP, PH_Vendor, 'Pi-hole'
# FROM PiHole_Network
# WHERE PH_LastQuery >= ?
# AND NOT EXISTS (SELECT 'X' FROM CurrentScan
# WHERE cur_MAC = PH_MAC
# AND cur_ScanCycle = ? )""",
# (cycle,
# (int(startTime.strftime('%s')) - 60 * p_cycle_interval),
# cycle) )
# ------------------------ TO CONVERT INTO PLUGIN
# Insert Pi-hole devices
startTime = timeNow()
sql.execute ("""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC,
cur_IP, cur_Vendor, cur_ScanMethod)
SELECT ?, PH_MAC, PH_IP, PH_Vendor, 'Pi-hole'
FROM PiHole_Network
WHERE PH_LastQuery >= ?
AND NOT EXISTS (SELECT 'X' FROM CurrentScan
WHERE cur_MAC = PH_MAC
AND cur_ScanCycle = ? )""",
(cycle,
(int(startTime.strftime('%s')) - 60 * p_cycle_interval),
cycle) )
# Check Internet connectivity # Check Internet connectivity
internet_IP = get_internet_IP( conf.DIG_GET_IP_ARG ) internet_IP = get_internet_IP( conf.DIG_GET_IP_ARG )
# TESTING - Force IP # TESTING - Force IP
# internet_IP = "" # internet_IP = ""
if internet_IP != "" : if internet_IP != "" :
sql.execute ("""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod) sql.execute (f"""INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod)
VALUES (?, 'Internet', ?, Null, 'queryDNS') """, (cycle, internet_IP) ) VALUES ( 1, 'Internet', '{internet_IP}', Null, 'queryDNS') """)
# #76 Add Local MAC of default local interface # #76 Add Local MAC of default local interface
# BUGFIX #106 - Device that pialert is running # BUGFIX #106 - Device that pialert is running
@@ -73,93 +77,67 @@ def save_scanned_devices (db, p_arpscan_devices, p_cycle_interval):
local_ip = '0.0.0.0' local_ip = '0.0.0.0'
# Check if local mac has been detected with other methods # Check if local mac has been detected with other methods
sql.execute ("SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanCycle = ? AND cur_MAC = ? ", (cycle, local_mac) ) sql.execute ("SELECT COUNT(*) FROM CurrentScan WHERE cur_MAC = ? ", (local_mac) )
if sql.fetchone()[0] == 0 : if sql.fetchone()[0] == 0 :
sql.execute ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod) "+ sql.execute ("INSERT INTO CurrentScan (cur_ScanCycle, cur_MAC, cur_IP, cur_Vendor, cur_ScanMethod) "+
"VALUES ( ?, ?, ?, Null, 'local_MAC') ", (cycle, local_mac, local_ip) ) "VALUES ( 1, ?, ?, Null, 'local_MAC') ", (local_mac, local_ip) )
#------------------------------------------------------------------------------- #-------------------------------------------------------------------------------
def print_scan_stats (db): def print_scan_stats (db):
sql = db.sql #TO-DO sql = db.sql #TO-DO
# Devices Detected # Devices Detected
sql.execute ("""SELECT COUNT(*) FROM CurrentScan sql.execute ("""SELECT COUNT(*) FROM CurrentScan""")
WHERE cur_ScanCycle = ? """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] Devices Detected.......: ', str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] Devices Detected.......: ', str (sql.fetchone()[0]) ])
# Devices arp-scan # Devices arp-scan
sql.execute ("""SELECT COUNT(*) FROM CurrentScan sql.execute ("""SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanMethod='arp-scan' """)
WHERE cur_ScanMethod='arp-scan' AND cur_ScanCycle = ? """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] arp-scan detected..: ', str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] arp-scan detected..: ', str (sql.fetchone()[0]) ])
# Devices Pi-hole # Devices Pi-hole
sql.execute ("""SELECT COUNT(*) FROM CurrentScan sql.execute ("""SELECT COUNT(*) FROM CurrentScan WHERE cur_ScanMethod='PiHole'""")
WHERE cur_ScanMethod='PiHole' AND cur_ScanCycle = ? """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] Pi-hole detected...: +' + str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] Pi-hole detected...: +' + str (sql.fetchone()[0]) ])
# New Devices # New Devices
sql.execute ("""SELECT COUNT(*) FROM CurrentScan sql.execute ("""SELECT COUNT(*) FROM CurrentScan
WHERE cur_ScanCycle = ? WHERE NOT EXISTS (SELECT 1 FROM Devices
AND NOT EXISTS (SELECT 1 FROM Devices WHERE dev_MAC = cur_MAC) """)
WHERE dev_MAC = cur_MAC) """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] New Devices........: ' + str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] New Devices........: ' + str (sql.fetchone()[0]) ])
# Devices in this ScanCycle
sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
AND dev_ScanCycle = ? """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] Devices in this cycle..: ' + str (sql.fetchone()[0]) ])
# Down Alerts # Down Alerts
sql.execute ("""SELECT COUNT(*) FROM Devices sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_AlertDeviceDown = 1 WHERE dev_AlertDeviceDown = 1
AND dev_ScanCycle = ?
AND NOT EXISTS (SELECT 1 FROM CurrentScan AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """)
(conf.cycle,))
mylog('verbose', ['[Scan Stats] Down Alerts........: ' + str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] Down Alerts........: ' + str (sql.fetchone()[0]) ])
# New Down Alerts # New Down Alerts
sql.execute ("""SELECT COUNT(*) FROM Devices sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_AlertDeviceDown = 1 WHERE dev_AlertDeviceDown = 1
AND dev_PresentLastScan = 1 AND dev_PresentLastScan = 1
AND dev_ScanCycle = ?
AND NOT EXISTS (SELECT 1 FROM CurrentScan AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """)
(conf.cycle,))
mylog('verbose', ['[Scan Stats] New Down Alerts....: ' + str (sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] New Down Alerts....: ' + str (sql.fetchone()[0]) ])
# New Connections # New Connections
sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
AND dev_PresentLastScan = 0 AND dev_PresentLastScan = 0""")
AND dev_ScanCycle = ? """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] New Connections....: ' + str ( sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] New Connections....: ' + str ( sql.fetchone()[0]) ])
# Disconnections # Disconnections
sql.execute ("""SELECT COUNT(*) FROM Devices sql.execute ("""SELECT COUNT(*) FROM Devices
WHERE dev_PresentLastScan = 1 WHERE dev_PresentLastScan = 1
AND dev_ScanCycle = ?
AND NOT EXISTS (SELECT 1 FROM CurrentScan AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """)
(conf.cycle,))
mylog('verbose', ['[Scan Stats] Disconnections.....: ' + str ( sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] Disconnections.....: ' + str ( sql.fetchone()[0]) ])
# IP Changes # IP Changes
sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan sql.execute ("""SELECT COUNT(*) FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
AND dev_ScanCycle = ? AND dev_LastIP <> cur_IP """)
AND dev_LastIP <> cur_IP """,
(conf.cycle,))
mylog('verbose', ['[Scan Stats] IP Changes.........: ' + str ( sql.fetchone()[0]) ]) mylog('verbose', ['[Scan Stats] IP Changes.........: ' + str ( sql.fetchone()[0]) ])
@@ -176,20 +154,18 @@ def create_new_devices (db):
eve_PendingAlertEmail) eve_PendingAlertEmail)
SELECT cur_MAC, cur_IP, ?, 'New Device', cur_Vendor, 1 SELECT cur_MAC, cur_IP, ?, 'New Device', cur_Vendor, 1
FROM CurrentScan FROM CurrentScan
WHERE cur_ScanCycle = ? WHERE NOT EXISTS (SELECT 1 FROM Devices
AND NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = cur_MAC) """, WHERE dev_MAC = cur_MAC) """,
(startTime, conf.cycle) ) (startTime) )
mylog('debug','[New Devices] Insert Connection into session table') mylog('debug','[New Devices] Insert Connection into session table')
sql.execute ("""INSERT INTO Sessions (ses_MAC, ses_IP, ses_EventTypeConnection, ses_DateTimeConnection, sql.execute ("""INSERT INTO Sessions (ses_MAC, ses_IP, ses_EventTypeConnection, ses_DateTimeConnection,
ses_EventTypeDisconnection, ses_DateTimeDisconnection, ses_StillConnected, ses_AdditionalInfo) ses_EventTypeDisconnection, ses_DateTimeDisconnection, ses_StillConnected, ses_AdditionalInfo)
SELECT cur_MAC, cur_IP,'Connected',?, NULL , NULL ,1, cur_Vendor SELECT cur_MAC, cur_IP,'Connected',?, NULL , NULL ,1, cur_Vendor
FROM CurrentScan FROM CurrentScan
WHERE cur_ScanCycle = ? WHERE NOT EXISTS (SELECT 1 FROM Sessions
AND NOT EXISTS (SELECT 1 FROM Sessions
WHERE ses_MAC = cur_MAC) """, WHERE ses_MAC = cur_MAC) """,
(startTime, conf.cycle) ) (startTime) )
# arpscan - Create new devices # arpscan - Create new devices
mylog('debug','[New Devices] 2 Create devices') mylog('debug','[New Devices] 2 Create devices')
@@ -236,13 +212,12 @@ def create_new_devices (db):
SELECT cur_MAC, '(unknown)', cur_Vendor, cur_IP, ?, ?, SELECT cur_MAC, '(unknown)', cur_Vendor, cur_IP, ?, ?,
{newDevDefaults} {newDevDefaults}
FROM CurrentScan FROM CurrentScan
WHERE cur_ScanCycle = ? WHERE NOT EXISTS (SELECT 1 FROM Devices
AND NOT EXISTS (SELECT 1 FROM Devices
WHERE dev_MAC = cur_MAC) """ WHERE dev_MAC = cur_MAC) """
mylog('debug',f'[New Devices] 2 Create devices SQL: {sqlQuery}') mylog('debug',f'[New Devices] 2 Create devices SQL: {sqlQuery}')
sql.execute (sqlQuery, (startTime, startTime, conf.cycle) ) sql.execute (sqlQuery, (startTime, startTime) )
# Pi-hole - Insert events for new devices # Pi-hole - Insert events for new devices
# NOT STRICYLY NECESARY (Devices can be created through Current_Scan) # NOT STRICYLY NECESARY (Devices can be created through Current_Scan)
@@ -326,18 +301,15 @@ def update_devices_data_from_scan (db):
WHERE dev_ScanCycle = ? WHERE dev_ScanCycle = ?
AND dev_PresentLastScan = 0 AND dev_PresentLastScan = 0
AND EXISTS (SELECT 1 FROM CurrentScan AND EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC) """,
AND dev_ScanCycle = cur_ScanCycle) """, (startTime))
(startTime, conf.cycle))
# Clean no active devices # Clean no active devices
mylog('debug','[Update Devices] 2 Clean no active devices') mylog('debug','[Update Devices] 2 Clean no active devices')
sql.execute ("""UPDATE Devices SET dev_PresentLastScan = 0 sql.execute ("""UPDATE Devices SET dev_PresentLastScan = 0
WHERE dev_ScanCycle = ? WHERE NOT EXISTS (SELECT 1 FROM CurrentScan
AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """)
(conf.cycle,))
# Update IP & Vendor # Update IP & Vendor
mylog('debug','[Update Devices] - 3 LastIP & Vendor') mylog('debug','[Update Devices] - 3 LastIP & Vendor')
@@ -348,11 +320,9 @@ def update_devices_data_from_scan (db):
dev_Vendor = (SELECT cur_Vendor FROM CurrentScan dev_Vendor = (SELECT cur_Vendor FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) AND dev_ScanCycle = cur_ScanCycle)
WHERE dev_ScanCycle = ? WHERE EXISTS (SELECT 1 FROM CurrentScan
AND EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """)
(conf.cycle,))
# Pi-hole Network - Update (unknown) Name # Pi-hole Network - Update (unknown) Name
mylog('debug','[Update Devices] - 4 Unknown Name') mylog('debug','[Update Devices] - 4 Unknown Name')

9
pialert/initialise.py
View File

@@ -59,16 +59,19 @@ def importConfigs (db):
# Only import file if the file was modifed since last import. # Only import file if the file was modifed since last import.
# this avoids time zone issues as we just compare the previous timestamp to the current time stamp # this avoids time zone issues as we just compare the previous timestamp to the current time stamp
fileModifiedTime = os.path.getmtime(config_file)
mylog('debug', ['[Import Config] checking config file ']) mylog('debug', ['[Import Config] checking config file '])
mylog('debug', ['[Import Config] lastImportedConfFile :', conf.lastImportedConfFile]) mylog('debug', ['[Import Config] lastImportedConfFile :', conf.lastImportedConfFile])
mylog('debug', ['[Import Config] file modified time :', os.path.getmtime(config_file)]) mylog('debug', ['[Import Config] file modified time :', fileModifiedTime])
if (os.path.getmtime(config_file) == conf.lastImportedConfFile) : if (fileModifiedTime == conf.lastImportedConfFile) :
mylog('debug', ['[Import Config] skipping config file import']) mylog('debug', ['[Import Config] skipping config file import'])
return return
conf.lastImportedConfFile = os.path.getmtime(config_file) conf.lastImportedConfFile = fileModifiedTime
mylog('debug', ['[Import Config] importing config file']) mylog('debug', ['[Import Config] importing config file'])
conf.mySettings = [] # reset settings conf.mySettings = [] # reset settings

40
pialert/networkscan.py
View File

@@ -36,12 +36,13 @@ def scan_network (db):
db.commitDB() db.commitDB()
# Moved to the ARPSCAN Plugin
# arp-scan command # arp-scan command
conf.arpscan_devices = [] # conf.arpscan_devices = []
if conf.ENABLE_ARPSCAN: # if conf.ENABLE_ARPSCAN:
mylog('verbose','[Network Scan] arp-scan start') # mylog('verbose','[Network Scan] arp-scan start')
conf.arpscan_devices = execute_arpscan (conf.userSubnets) # conf.arpscan_devices = execute_arpscan (conf.userSubnets)
mylog('verbose','[Network Scan] arp-scan ends') # mylog('verbose','[Network Scan] arp-scan ends')
# Pi-hole method # Pi-hole method
if conf.PIHOLE_ACTIVE : if conf.PIHOLE_ACTIVE :
@@ -57,8 +58,7 @@ def scan_network (db):
def process_scan (db, arpscan_devices): def process_scan (db):
# Query ScanCycle properties # Query ScanCycle properties
scanCycle_data = query_ScanCycle_Data (db, True) scanCycle_data = query_ScanCycle_Data (db, True)
@@ -76,7 +76,7 @@ def process_scan (db, arpscan_devices):
# Load current scan data # Load current scan data
mylog('verbose','[Process Scan] Processing scan results') mylog('verbose','[Process Scan] Processing scan results')
save_scanned_devices (db, arpscan_devices, cycle_interval) save_scanned_devices (db)
db.commitDB() db.commitDB()
@@ -85,8 +85,7 @@ def process_scan (db, arpscan_devices):
print_scan_stats(db) print_scan_stats(db)
mylog('none','[Process Scan] Stats end') mylog('none','[Process Scan] Stats end')
# Create Events # Create Events
mylog('verbose','[Process Scan] Updating DB Info')
mylog('verbose','[Process Scan] Sessions Events (connect / discconnect)') mylog('verbose','[Process Scan] Sessions Events (connect / discconnect)')
insert_events(db) insert_events(db)
@@ -122,6 +121,9 @@ def process_scan (db, arpscan_devices):
# Skip repeated notifications # Skip repeated notifications
mylog('verbose','[Process Scan] Skipping repeated notifications') mylog('verbose','[Process Scan] Skipping repeated notifications')
skip_repeated_notifications (db) skip_repeated_notifications (db)
# Clear current scan as processed
db.sql.execute ("DELETE FROM CurrentScan")
# Commit changes # Commit changes
db.commitDB() db.commitDB()
@@ -279,12 +281,11 @@ def insert_events (db):
SELECT dev_MAC, dev_LastIP, ?, 'Device Down', '', 1 SELECT dev_MAC, dev_LastIP, ?, 'Device Down', '', 1
FROM Devices FROM Devices
WHERE dev_AlertDeviceDown = 1 WHERE dev_AlertDeviceDown = 1
AND dev_PresentLastScan = 1 AND dev_PresentLastScan = 1
AND dev_ScanCycle = ?
AND NOT EXISTS (SELECT 1 FROM CurrentScan AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """,
(startTime, conf.cycle) ) (startTime) )
# Check new connections # Check new connections
mylog('debug','[Events] - 2 - New Connections') mylog('debug','[Events] - 2 - New Connections')
@@ -294,9 +295,8 @@ def insert_events (db):
SELECT cur_MAC, cur_IP, ?, 'Connected', '', dev_AlertEvents SELECT cur_MAC, cur_IP, ?, 'Connected', '', dev_AlertEvents
FROM Devices, CurrentScan FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
AND dev_PresentLastScan = 0 AND dev_PresentLastScan = 0 """,
AND dev_ScanCycle = ? """, (startTime) )
(startTime, conf.cycle) )
# Check disconnections # Check disconnections
mylog('debug','[Events] - 3 - Disconnections') mylog('debug','[Events] - 3 - Disconnections')
@@ -308,11 +308,10 @@ def insert_events (db):
FROM Devices FROM Devices
WHERE dev_AlertDeviceDown = 0 WHERE dev_AlertDeviceDown = 0
AND dev_PresentLastScan = 1 AND dev_PresentLastScan = 1
AND dev_ScanCycle = ?
AND NOT EXISTS (SELECT 1 FROM CurrentScan AND NOT EXISTS (SELECT 1 FROM CurrentScan
WHERE dev_MAC = cur_MAC WHERE dev_MAC = cur_MAC
AND dev_ScanCycle = cur_ScanCycle) """, AND dev_ScanCycle = cur_ScanCycle) """,
(startTime, conf.cycle) ) (startTime) )
# Check IP Changed # Check IP Changed
mylog('debug','[Events] - 4 - IP Changes') mylog('debug','[Events] - 4 - IP Changes')
@@ -322,8 +321,7 @@ def insert_events (db):
SELECT cur_MAC, cur_IP, ?, 'IP Changed', SELECT cur_MAC, cur_IP, ?, 'IP Changed',
'Previous IP: '|| dev_LastIP, dev_AlertEvents 'Previous IP: '|| dev_LastIP, dev_AlertEvents
FROM Devices, CurrentScan FROM Devices, CurrentScan
WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle WHERE dev_MAC = cur_MAC AND dev_ScanCycle = cur_ScanCycle
AND dev_ScanCycle = ?
AND dev_LastIP <> cur_IP """, AND dev_LastIP <> cur_IP """,
(startTime, conf.cycle) ) (startTime) )
mylog('debug','[Events] - Events end') mylog('debug','[Events] - Events end')

47
pialert/plugin.py
View File

@@ -181,8 +181,8 @@ def execute_plugin(db, plugin):
# build SQL query parameters to insert into the DB # build SQL query parameters to insert into the DB
sqlParams = [] sqlParams = []
# python-script # script
if plugin['data_source'] == 'python-script': if plugin['data_source'] == 'script':
# ------- prepare params -------- # ------- prepare params --------
# prepare command from plugin settings, custom parameters # prepare command from plugin settings, custom parameters
command = resolve_wildcards_arr(set_CMD.split(), params) command = resolve_wildcards_arr(set_CMD.split(), params)
@@ -203,24 +203,35 @@ def execute_plugin(db, plugin):
# check the last run output # check the last run output
f = open(pluginsPath + '/' + plugin["code_name"] + '/last_result.log', 'r+') # Initialize newLines
newLines = f.read().split('\n') newLines = []
f.close()
# cleanup - select only lines containing a separator to filter out unnecessary data # Create the file path
newLines = list(filter(lambda x: '|' in x, newLines)) file_path = os.path.join(pluginsPath, plugin["code_name"], 'last_result.log')
# # regular logging # Check if the file exists
# for line in newLines: if os.path.exists(file_path):
# append_line_to_file (pluginsPath + '/plugin.log', line +'\n') # File exists, open it and read its contents
with open(file_path, 'r+') as f:
for line in newLines: newLines = f.read().split('\n')
columns = line.split("|")
# There has to be always 9 columns # if the script produced some outpout, clean it up to ensure it's the correct format
if len(columns) == 9: # cleanup - select only lines containing a separator to filter out unnecessary data
sqlParams.append((plugin["unique_prefix"], columns[0], columns[1], 'null', columns[2], columns[3], columns[4], columns[5], columns[6], 0, columns[7], 'null', columns[8])) newLines = list(filter(lambda x: '|' in x, newLines))
else:
mylog('none', ['[Plugins]: Skipped invalid line in the output: ', line]) # # regular logging
# for line in newLines:
# append_line_to_file (pluginsPath + '/plugin.log', line +'\n')
for line in newLines:
columns = line.split("|")
# There has to be always 9 columns
if len(columns) == 9:
sqlParams.append((plugin["unique_prefix"], columns[0], columns[1], 'null', columns[2], columns[3], columns[4], columns[5], columns[6], 0, columns[7], 'null', columns[8]))
else:
mylog('none', ['[Plugins]: Skipped invalid line in the output: ', line])
else:
mylog('debug', [f'[Plugins] The file {file_path} does not exist'])
# pialert-db-query # pialert-db-query
if plugin['data_source'] == 'pialert-db-query': if plugin['data_source'] == 'pialert-db-query':