docs + convert pholus to plugin v0.1

This commit is contained in:
Jokob-sk
2023-08-24 08:03:19 +10:00
parent 5103e2c229
commit 3b60a3a1ae
8 changed files with 650 additions and 5 deletions

View File

@@ -78,7 +78,7 @@ There are 2 approaches how to get PiHole devices imported. Via the PiHole import
* `DHCPLSS_RUN`: You need to map `:/etc/pihole/dhcp.leases` in the `docker-compose.yml` file if you enable this setting.
* The above setting has to be matched with a corresponding `DHCPLSS_paths_to_check` setting entry (the path in the container must contain `pihole` as PiHole uses a different format of the `dhcp.leases` file).
> It's recommended to use the same schedule interval for all plugins scanning your network.
> It's recommended to use the same schedule interval for all plugins responsible for discovering new devices.
### **Common issues**

View File

@@ -1,7 +1,8 @@
## 📚 Docs for individual plugins
[Click here, if you want to see the Spanish version
](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/README_ES.md)
### Community translations of this file
- [Spanish](https://github.com/jokob-sk/Pi.Alert/blob/main/front/plugins/README_ES.md)
### Script based plugins

0
front/plugins/README_ES.md Normal file → Executable file
View File

View File

@@ -150,7 +150,7 @@
}],
"description": [{
"language_code":"en_us",
"string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#ARPSCAN_RUN\"><code>ARPSCAN_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>*/3 * * * *</code> will run the scan every 3 minutes. Will be run NEXT time the time passes. <br/> It's recommended to use the same schedule interval for all plugins scanning your network."
"string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#ARPSCAN_RUN\"><code>ARPSCAN_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>*/3 * * * *</code> will run the scan every 3 minutes. Will be run NEXT time the time passes. <br/> It's recommended to use the same schedule interval for all plugins responsible for discovering new devices."
},
{
"language_code":"es_es",

View File

@@ -0,0 +1,27 @@
## Overview
A plugin allowing for importing Un-Discoverable devices from the settings page.
The main usecase is to add dumb network gear like unmanaged hubs and switches to the network view.
There might be other usecases, please let me know.
### Usage
- Go to settings and find Un-Discoverabe Devices in the list of plugins.
- Enable the plugin by changing the RUN parameter from disabled to `once` or `always_after_scan`.
- Add the name of your device to the list. (remove the sample entry first)
- SAVE
- wait for the next scan to finish
#### Examples:
Settings:
![settings](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/52883307-19a5-4602-b13a-9825461f6cc4)
resulting in these devices:
![devices](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/9f7659e7-75a8-4ae9-9f5f-781bdbcbc949)
Allowing Un-Discoverable devices like hubs, switches or APs to be added to the network view.
![network](https://github.com/Data-Monkey/Pi.Alert/assets/7224371/b5ccc3b3-f5fd-4f5b-b0f0-e4e637c6da33)
### Known Limitations
- Un-Discoverable Devices always show as offline. That is expected as they can not be discovered by Pi.Alert.
- All IPs are set to 0.0.0.0 therefore the "Random MAC" icon might show up.

View File

@@ -0,0 +1,339 @@
{
"code_name": "pholus_scan",
"unique_prefix": "PHOLUS",
"enabled": true,
"data_source": "script",
"mapped_to_table": "Pholus_Scan",
"data_filters": [
{
"compare_column" : "Object_PrimaryID",
"compare_operator" : "==",
"compare_field_id": "txtMacFilter",
"compare_js_template": "'{value}'.toString()",
"compare_use_quotes": true
}
],
"show_ui": true,
"localized": ["display_name", "description", "icon"],
"display_name": [
{
"language_code": "en_us",
"string": "Pholus-Scan (Name discovery)"
}
],
"icon": [
{
"language_code": "en_us",
"string": "<i class=\"fa-solid fa-search\"></i>"
},
{
"language_code": "es_es",
"string": "<i class=\"fa-solid fa-search\"></i>"
}
],
"description": [
{
"language_code": "en_us",
"string": "This plugin is to execute a Pholus-scan (name discovery) on the local network"
}
],
"params" : [
{
"name" : "subnets",
"type" : "setting",
"value" : "SCAN_SUBNETS"
},
{
"name" : "timeout",
"type" : "setting",
"value" : "RUN_TIMEOUT"
}
],
"settings": [
{
"function": "RUN",
"type": "text.select",
"default_value":"schedule",
"options": ["disabled", "once", "schedule", "always_after_scan", "on_new_device"],
"localized": ["name", "description"],
"events": ["run"],
"name" :[
{
"language_code":"en_us",
"string" : "When to run"
},
{
"language_code":"es_es",
"string" : "Cuando ejecutar"
}],
"description": [{
"language_code":"en_us",
"string" : "Specify when your Name-discovery scan will run. Typical setting would be <code>on_new_device</code> or <code>schedule</code> and then you specify a cron-like schedule in the <a href=\"#PHOLUS_RUN_SCHD\"><code>PHOLUS_RUN_SCHD</code>setting</a>."
}]
},
{
"function": "CMD",
"type": "readonly",
"default_value": "python3 /home/pi/pialert/front/plugins/pholus_scan/script.py userSubnets={subnets} timeoutSec={timeout}",
"options": [],
"localized": ["name", "description"],
"name": [
{
"language_code": "en_us",
"string": "Command"
},
{
"language_code": "es_es",
"string": "Comando"
}
],
"description": [
{
"language_code": "en_us",
"string": "Command to run. This should not be changed"
},
{
"language_code": "es_es",
"string": "Comando para ejecutar. Esto no debe ser cambiado"
}
]
},
{
"function": "RUN_TIMEOUT",
"type": "integer",
"default_value": 300,
"options": [],
"localized": ["name", "description"],
"name": [
{
"language_code": "en_us",
"string": "Run timeout"
},
{
"language_code": "es_es",
"string": "Tiempo límite de ejecución"
}
],
"description": [
{
"language_code": "en_us",
"string": "Network scan time in seconds. Pholus scan will always run this long. The longer it runs the more device names might be resolved. Will be divided by the number of subnets."
}
]
},
{
"function": "RUN_SCHD",
"type": "text",
"default_value":"30 3 * * *",
"options": [],
"localized": ["name", "description"],
"name" : [{
"language_code":"en_us",
"string" : "Schedule"
},
{
"language_code":"es_es",
"string" : "Schedule"
}],
"description": [{
"language_code":"en_us",
"string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#PHOLUS_RUN\"><code>PHOLUS_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>30 3 * * *</code> will run the scan at 3:30 am. Will be run NEXT time the time passes. <br/>"
}]
},
{
"function": "WATCH",
"type": "text.multiselect",
"default_value":["Watched_Value1", "Watched_Value2"],
"options": ["Watched_Value1","Watched_Value2","Watched_Value3","Watched_Value4"],
"localized": ["name", "description"],
"name" :[{
"language_code":"en_us",
"string" : "Watched"
},
{
"language_code":"es_es",
"string" : "Watched"
}] ,
"description":[{
"language_code":"en_us",
"string" : "Send a notification if selected values change. Use <code>CTRL + Click</code> to select/deselect. <ul> <li><code>Watched_Value1</code> is TBC</li><li><code>Watched_Value2</code> is TBC</li><li><code>Watched_Value3</code> is TBC </li><li><code>Watched_Value4</code> is TBC </li></ul>"
}]
},
{
"function": "REPORT_ON",
"type": "text.multiselect",
"default_value": ["new"],
"options": ["new", "watched-changed", "watched-not-changed", "missing-in-last-scan"],
"localized": ["name", "description"],
"name": [
{
"language_code": "en_us",
"string": "Report on"
},
{
"language_code": "es_es",
"string": "Informar sobre"
}
],
"description": [
{
"language_code": "en_us",
"string": "When should notification be sent out."
},
{
"language_code": "es_es",
"string": "Cuándo debe enviarse una notificación."
}
]
}
],
"database_column_definitions":
[
{
"column": "Object_PrimaryID",
"mapped_to_column": "cur_MAC",
"css_classes": "col-sm-2",
"show": true,
"type": "device_name_mac",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "MAC"
},
{
"language_code":"es_es",
"string" : "MAC"
}]
},
{
"column": "Watched_Value1",
"mapped_to_column": "cur_IP",
"css_classes": "col-sm-2",
"show": true,
"type": "device_ip",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "IP"
},
{
"language_code":"es_es",
"string" : "IP"
}]
},
{
"column": "Watched_Value2",
"mapped_to_column": "cur_Vendor",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Vendor"
},
{
"language_code":"es_es",
"string" : "Proveedor"
}]
} ,
{
"column": "Extra",
"mapped_to_column": "cur_ScanMethod",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Scan method"
},
{
"language_code":"es_es",
"string" : "Método de escaneado"
}]
} ,
{
"column": "DateTimeCreated",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Created"
},
{
"language_code":"es_es",
"string" : "Creado"
}]
},
{
"column": "DateTimeChanged",
"css_classes": "col-sm-2",
"show": true,
"type": "label",
"default_value":"",
"options": [],
"localized": ["name"],
"name":[
{
"language_code":"en_us",
"string" : "Changed"
},
{
"language_code":"es_es",
"string" : "Cambiado"
}
]
},
{
"column": "Status",
"css_classes": "col-sm-1",
"show": true,
"type": "replace",
"default_value":"",
"options": [
{
"equals": "watched-not-changed",
"replacement": "<div style='text-align:center'><i class='fa-solid fa-square-check'></i><div></div>"
},
{
"equals": "watched-changed",
"replacement": "<div style='text-align:center'><i class='fa-solid fa-triangle-exclamation'></i></div>"
},
{
"equals": "new",
"replacement": "<div style='text-align:center'><i class='fa-solid fa-circle-plus'></i></div>"
},
{
"equals": "missing-in-last-scan",
"replacement": "<div style='text-align:center'><i class='fa-solid fa-question'></i></div>"
}
],
"localized": ["name"],
"name":[{
"language_code":"en_us",
"string" : "Status"
},
{
"language_code":"es_es",
"string" : "Estado"
}]
}
]
}

View File

@@ -0,0 +1,278 @@
#!/usr/bin/env python
import os
import pathlib
import argparse
import sys
import re
import base64
import subprocess
from time import strftime
sys.path.append("/home/pi/pialert/front/plugins")
sys.path.append('/home/pi/pialert/pialert')
from logger import mylog
from plugin_helper import Plugin_Object, Plugin_Objects
from const import fullPholusPath, logPath
CUR_PATH = str(pathlib.Path(__file__).parent.resolve())
LOG_FILE = os.path.join(CUR_PATH, 'script.log')
RESULT_FILE = os.path.join(CUR_PATH, 'last_result.log')
def main():
# sample
# /home/pi/pialert/front/plugins/pholus_scan/script.py userSubnets=b'MTkyLjE2OC4xLjAvMjQgLS1pbnRlcmZhY2U9ZXRoMQ==' timeoutSec=10
# sudo docker exec pialert /home/pi/pialert/front/plugins/pholus_scan__ignore/script.py userSubnets=b'MTkyLjE2OC4xLjAvMjQgLS1pbnRlcmZhY2U9ZXRoMQ==' timeoutSec=10
# the script expects a parameter in the format of userSubnets=subnet1,subnet2,...
parser = argparse.ArgumentParser(description='Import devices from settings')
parser.add_argument('userSubnets', nargs='+', help="list of subnets with options")
parser.add_argument('timeoutSec', nargs='+', help="timeout")
values = parser.parse_args()
# Assuming Plugin_Objects is a class or function that reads data from the RESULT_FILE
# and returns a list of objects called 'devices'.
plug_objects = Plugin_Objects(RESULT_FILE)
# Print a message to indicate that the script is starting.
print('In script:')
# Assuming 'values' is a dictionary or object that contains a key 'userSubnets'
# which holds a list of user-submitted subnets.
# Printing the userSubnets list to check its content.
print(values.userSubnets)
# Extract the base64-encoded subnet information from the first element of the userSubnets list.
# The format of the element is assumed to be like 'userSubnets=b<base64-encoded-data>'.
userSubnetsParamBase64 = values.userSubnets[0].split('userSubnets=b')[1]
timeoutSec = values.timeoutSec[0].split('=')[1]
# Printing the extracted base64-encoded subnet information.
print(userSubnetsParamBase64)
print(timeoutSec)
# Decode the base64-encoded subnet information to get the actual subnet information in ASCII format.
userSubnetsParam = base64.b64decode(userSubnetsParamBase64).decode('ascii')
# Print the decoded subnet information.
print('userSubnetsParam:')
print(userSubnetsParam)
# Check if the decoded subnet information contains multiple subnets separated by commas.
# If it does, split the string into a list of individual subnets.
# Otherwise, create a list with a single element containing the subnet information.
if ',' in userSubnetsParam:
subnets_list = userSubnetsParam.split(',')
else:
subnets_list = [userSubnetsParam]
# Execute the ARP scanning process on the list of subnets (whether it's one or multiple subnets).
# The function 'execute_arpscan' is assumed to be defined elsewhere in the code.
all_entries = execute_pholus_scan(subnets_list, timeoutSec)
for entry in all_entries:
plug_objects.add_object(
# "Info", "Time", "MAC", "IP_v4_or_v6", "Record_Type", "Value"
primaryId = entry[2],
secondaryId = entry[3],
watched1 = entry[0],
watched2 = entry[4],
watched3 = entry[5],
watched4 = '',
extra = entry[0],
foreignKey = entry[2])
plug_objects.write_result_file()
return 0
def execute_pholus_scan(userSubnets, timeoutSec):
# output of possible multiple interfaces
arpscan_output = ""
result_list = []
timeoutPerSubnet = float(timeoutSec) / len(userSubnets)
print(timeoutPerSubnet)
# scan each interface
for interface in userSubnets:
temp = interface.split("--interface=")
if len(temp) != 2:
mylog('none', ["[PholusScan] Skip scan (need interface in format '192.168.1.0/24 --inteface=eth0'), got: ", interface])
return
mask = temp[0].strip()
interface = temp[1].strip()
pholus_output_list = execute_pholus_on_interface (interface, timeoutPerSubnet, mask)
print(pholus_output_list)
result_list += pholus_output_list
print("List len:", len(result_list))
print("List:", result_list)
return result_list
def execute_pholus_on_interface(interface, timeoutSec, mask):
# logging & updating app state
mylog('none', ['[PholusScan] Scan: Pholus for ', str(timeoutSec), 's ('+ str(round(int(timeoutSec) / 60, 1)) +'min)'])
mylog('verbose', ["[PholusScan] Pholus scan on [interface] ", interface, " [mask] " , mask])
# the scan always lasts 2x as long, so the desired user time from settings needs to be halved
adjustedTimeout = str(round(int(timeoutSec) / 2, 0))
# python3 -m trace --trace /home/pi/pialert/pholus/pholus3.py eth1 -rdns_scanning 192.168.1.0/24 -stimeout 600
pholus_args = ['python3', fullPholusPath, interface, "-rdns_scanning", mask, "-stimeout", adjustedTimeout]
# Execute command
output = ""
try:
# try runnning a subprocess with a forced (timeout + 30 seconds) in case the subprocess hangs
output = subprocess.check_output (pholus_args, universal_newlines=True, stderr=subprocess.STDOUT, timeout=(timeoutSec + 30))
except subprocess.CalledProcessError as e:
# An error occured, handle it
mylog('none', ['[PholusScan]', e.output])
mylog('none', ["[PholusScan] Error - Pholus Scan - check logs"])
except subprocess.TimeoutExpired as timeErr:
mylog('none', ['[PholusScan] Pholus TIMEOUT - the process forcefully terminated as timeout reached'])
if output == "": # check if the subprocess failed
mylog('none', ['[PholusScan] Scan: Pholus FAIL - check logs'])
else:
mylog('verbose', ['[PholusScan] Scan: Pholus SUCCESS'])
# check the last run output
f = open(CUR_PATH + '/pialert_pholus_lastrun.log', 'r+')
newLines = f.read().split('\n')
f.close()
# cleanup - select only lines containing a separator to filter out unnecessary data
newLines = list(filter(lambda x: '|' in x, newLines))
# build SQL query parameters to insert into the DB
params = []
for line in newLines:
columns = line.split("|")
if len(columns) == 4:
# "Info", "Time", "MAC", "IP_v4_or_v6", "Record_Type", "Value"
params.append( interface + " " + mask, timeNowTZ() , columns[0].replace(" ", ""), columns[1].replace(" ", ""), columns[2].replace(" ", ""), columns[3])
return params
#-------------------------------------------------------------------------------
def cleanResult(str):
# alternative str.split('.')[0]
str = str.replace("._airplay", "")
str = str.replace("._tcp", "")
str = str.replace(".local", "")
str = str.replace("._esphomelib", "")
str = str.replace("._googlecast", "")
str = str.replace(".lan", "")
str = str.replace(".home", "")
str = re.sub(r'-[a-fA-F0-9]{32}', '', str) # removing last part of e.g. Nest-Audio-ff77ff77ff77ff77ff77ff77ff77ff77
str = re.sub(r'#.*', '', str) # Remove everything after '#' including the '#'
# remove trailing dots
if str.endswith('.'):
str = str[:-1]
return str
# Disclaimer - I'm interfacing with a script I didn't write (pholus3.py) so it's possible I'm missing types of answers
# it's also possible the pholus3.py script can be adjusted to provide a better output to interface with it
# Hit me with a PR if you know how! :)
def resolve_device_name_pholus (pMAC, pIP, allRes):
pholusMatchesIndexes = []
index = 0
for result in allRes:
# limiting entries used for name resolution to the ones containing the current IP (v4 only)
if result["MAC"] == pMAC and result["Record_Type"] == "Answer" and result["IP_v4_or_v6"] == pIP and '._googlezone' not in result["Value"]:
# found entries with a matching MAC address, let's collect indexes
pholusMatchesIndexes.append(index)
index += 1
# return if nothing found
if len(pholusMatchesIndexes) == 0:
return -1
# we have some entries let's try to select the most useful one
# airplay matches contain a lot of information
# Matches for example:
# Brand Tv (50)._airplay._tcp.local. TXT Class:32769 "acl=0 deviceid=66:66:66:66:66:66 features=0x77777,0x38BCB46 rsf=0x3 fv=p20.T-FFFFFF-03.1 flags=0x204 model=XXXX manufacturer=Brand serialNumber=XXXXXXXXXXX protovers=1.1 srcvers=777.77.77 pi=FF:FF:FF:FF:FF:FF psi=00000000-0000-0000-0000-FFFFFFFFFF gid=00000000-0000-0000-0000-FFFFFFFFFF gcgl=0 pk=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and '._airplay._tcp.local. TXT Class:32769' in str(allRes[i]["Value"]) :
return allRes[i]["Value"].split('._airplay._tcp.local. TXT Class:32769')[0]
# second best - contains airplay
# Matches for example:
# _airplay._tcp.local. PTR Class:IN "Brand Tv (50)._airplay._tcp.local."
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and '_airplay._tcp.local. PTR Class:IN' in allRes[i]["Value"] and ('._googlecast') not in allRes[i]["Value"]:
return cleanResult(allRes[i]["Value"].split('"')[1])
# Contains PTR Class:32769
# Matches for example:
# 3.1.168.192.in-addr.arpa. PTR Class:32769 "MyPc.local."
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'PTR Class:32769' in allRes[i]["Value"]:
return cleanResult(allRes[i]["Value"].split('"')[1])
# Contains AAAA Class:IN
# Matches for example:
# DESKTOP-SOMEID.local. AAAA Class:IN "fe80::fe80:fe80:fe80:fe80"
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'AAAA Class:IN' in allRes[i]["Value"]:
return cleanResult(allRes[i]["Value"].split('.local.')[0])
# Contains _googlecast._tcp.local. PTR Class:IN
# Matches for example:
# _googlecast._tcp.local. PTR Class:IN "Nest-Audio-ff77ff77ff77ff77ff77ff77ff77ff77._googlecast._tcp.local."
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and '_googlecast._tcp.local. PTR Class:IN' in allRes[i]["Value"] and ('Google-Cast-Group') not in allRes[i]["Value"]:
return cleanResult(allRes[i]["Value"].split('"')[1])
# Contains A Class:32769
# Matches for example:
# Android.local. A Class:32769 "192.168.1.6"
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and ' A Class:32769' in allRes[i]["Value"]:
return cleanResult(allRes[i]["Value"].split(' A Class:32769')[0])
# # Contains PTR Class:IN
# Matches for example:
# _esphomelib._tcp.local. PTR Class:IN "ceiling-light-1._esphomelib._tcp.local."
for i in pholusMatchesIndexes:
if checkIPV4(allRes[i]['IP_v4_or_v6']) and 'PTR Class:IN' in allRes[i]["Value"]:
if allRes[i]["Value"] and len(allRes[i]["Value"].split('"')) > 1:
return cleanResult(allRes[i]["Value"].split('"')[1])
return -1
#===============================================================================
# BEGIN
#===============================================================================
if __name__ == '__main__':
main()

View File

@@ -139,7 +139,7 @@
}],
"description": [{
"language_code":"en_us",
"string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#ARPSCAN_RUN\"><code>ARPSCAN_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>*/30 * * * *</code> will run the scan every 30 minutes. Will be run NEXT time the time passes. <br/> It's recommended to use the same schedule interval for all plugins scanning your network."
"string" : "Only enabled if you select <code>schedule</code> in the <a href=\"#ARPSCAN_RUN\"><code>ARPSCAN_RUN</code> setting</a>. Make sure you enter the schedule in the correct cron-like format (e.g. validate at <a href=\"https://crontab.guru/\" target=\"_blank\">crontab.guru</a>). For example entering <code>*/30 * * * *</code> will run the scan every 30 minutes. Will be run NEXT time the time passes. <br/> It's recommended to use the same schedule interval for all plugins responsible for discovering new devices."
},
{
"language_code":"es_es",