Fix debounce of api points to address Disk IO #914 + NMAPDEV_FAKE_MAC

docs/REMOTE_NETWORKS.md

@@ -0,0 +1,45 @@
+# Scanning Remote or Inaccessible Networks
+
+By design, local network scanners such as `arp-scan` use ARP (Address Resolution Protocol) to map IP addresses to MAC addresses on the local network. Since ARP operates at Layer 2 (Data Link Layer), it typically works only within a single broadcast domain, usually limited to a single router or network segment.
+
+## Complex Use Cases
+
+The following network setups might make some devices undetectable. Check the specific setup to understand the cause and find potential workarounds to still report on these devices.
+
+### Wi-Fi Extenders
+
+Wi-Fi extenders typically create a separate network or subnet, which can prevent network scanning tools like `arp-scan` from detecting devices behind the extender.
+
+> **Possible workaround**: Scan the specific subnet that the extender uses, if it is separate from the main network.
+
+### VPNs
+
+ARP operates at Layer 2 (Data Link Layer) and works only within a local area network (LAN). VPNs, which operate at Layer 3 (Network Layer), route traffic between networks, preventing ARP requests from discovering devices outside the local network.
+
+VPNs use virtual interfaces (e.g., `tun0`, `tap0`) to encapsulate traffic, bypassing ARP-based discovery. Additionally, many VPNs use NAT, which masks individual devices behind a shared IP address.
+
+> **Possible workaround**: Configure the VPN to bridge networks instead of routing to enable ARP, though this depends on the VPN setup and security requirements.
+
+# Other Workarounds
+
+The following workarounds should work for most complex network setups.
+
+## Supplementing Plugins
+
+You can use supplementary plugins that employ alternate methods. Protocols used by the `SNMPDSC` or `DHCPLSS` plugins are widely supported on different routers and can be effective as workarounds. Check the [plugins list](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/README.md) to find a plugin that works with your router and network setup.
+
+## Multiple NetAlertX Instances
+
+If you have servers in different networks, you can set up separate NetAlertX instances on those subnets and synchronize the results into one instance using the [`SYNC` plugin](https://github.com/jokob-sk/NetAlertX/tree/main/front/plugins/sync).
+
+## Manual Entry
+
+If you don't need to discover new devices and only need to report on their status (`online`, `offline`, `down`), you can manually enter devices and check their status using the [`ICMP` plugin](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/icmp_scan/), which uses the `ping` command internally.
+
+For more information on how to add devices manually (or dummy devices), refer to the [Device Management](https://github.com/jokob-sk/NetAlertX/blob/main/docs/DEVICE_MANAGEMENT.md) documentation.
+
+To create truly dummy devices, you can use a loopback IP address (e.g., `0.0.0.0` or `127.0.0.1`) so they appear online.
+
+## NMAP and Fake MAC Addresses
+
+Scanning remote networks with NMAP is possible, but since it cannot retrieve the MAC address, you need to enable the `NMAPDEV_FAKE_MAC` setting. This will generate a fake MAC address based on the IP address, allowing you to track devices. However, this can lead to inconsistencies, especially if the IP address changes or a previously logged device is rediscovered. If this setting is disabled, only the IP address will be discovered, and devices with missing MAC addresses will be skipped.

docs/SUBNETS.md

@@ -8,9 +8,7 @@ You need to specify the network interface and the network mask. You can also con
 
 In this example, `--interface=eth0 192.168.1.0/24` represents a neighboring subnet. If this command returns no results, the network is not accessible due to your network or firewall restrictions.
 
-If direct scans are not possible, you can use [supplementing plugins](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/README.md) that use alternate methods. Protocols used by the `SNMPDSC` or `DHCPLSS` plugins have good support and usually can be used as a workaround.
+If direct scans are not possible (Wi-Fi Extenders, VPNs and inaccessible networks), check the remote [networks documentation](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/REMOTE_NETWORKS.md). 
-
-Alternatively, you can set up separate NetAlertX instances running on the subnets and synchronize the results into one instance with the [`SYNC` plugin](https://github.com/jokob-sk/NetAlertX/tree/main/front/plugins/sync).
 
 > [!TIP] 
 > You may need to increase the time between scans `ARPSCAN_RUN_SCHD` and the timeout `ARPSCAN_RUN_TIMEOUT` (and similar settings for related plugins) when adding more subnets. If the timeout setting is exceeded, the scan is canceled to prevent the application from hanging due to rogue plugins.  
@@ -111,20 +109,3 @@ Please note the accessibility of macvlans when configured on the same computer.
 - NetAlertX does not detect the macvlan container when it is running on the same computer.
 - NetAlertX recognizes the macvlan container when it is running on a different computer.
 
-
-### Wi-Fi Extenders
-
-A Wi-Fi extender typically works by creating a separate network or subnet, which can cause certain network scanning tools, like `arp-scan`, to be unable to detect devices behind the extender.
-
-This happens because `arp-scan` uses ARP (Address Resolution Protocol) to map IP addresses to MAC addresses on the local network. Since ARP is a Layer 2 (data link layer) protocol, it usually only works within a single broadcast domain, which is typically limited to a single router or network segment.
-
-When you introduce a Wi-Fi extender, it may isolate devices on different segments of the network, meaning ARP packets cannot easily traverse from one segment (your main network) to another (the network behind the extender).
-
-To scan devices behind the extender, you can try:
-
-- Scanning the specific subnet that the extender uses, if it is separate from the main network.
-- Using [supplementing plugins](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/README.md) that use alternate methods. Protocols used by the `SNMPDSC` or `DHCPLSS` plugins have good support and usually can be used as a workaround.
-
-Check the [plugins list](https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/README.md) to find a plugin supported by your router and  your network setup. 
-
-

front/plugins/nmap_dev_scan/config.json

@@ -340,6 +340,34 @@
           "string": "Arguments to run nmap-scan with. Recommended and tested only with the setting: <br/> <code>sudo nmap -sn -PR -oX - </code>. <br/><br/> Note: The plugin attaches the interface and network mask, for example <code> -e eth1 192.168.1.0/24</code> and performs a separate scan for each interface specified in the <a onclick=\"toggleAllSettings()\" href=\"#SCAN_SUBNETS\"><code>SCAN_SUBNETS</code> setting</a>."
         }
       ]
+    },
+    {
+      "function": "FAKE_MAC",
+      "type": {
+        "dataType": "boolean",
+        "elements": [
+          {
+            "elementType": "input",
+            "elementOptions": [{ "type": "checkbox" }],
+            "transformers": []
+          }
+        ]
+      },
+      "default_value": false,
+      "options": [],
+      "localized": ["name", "description"],
+      "name": [
+        {
+          "language_code": "en_us",
+          "string": "Fake MAC if empty"
+        }
+      ],
+      "description": [
+        {
+          "language_code": "en_us",
+          "string": "When scanning remote networks, NMAP can only retrieve the IP address, not the MAC address. Enabling this setting generates a fake MAC address from the IP address to track devices, but it may cause inconsistencies if IPs change or devices are rediscovered. Static IPs are recommended. Device type and icon will not be detected correctly. When unchecked, devices with empty MAC addresses are skipped."
+        }
+      ]
     }
   ],
   "database_column_definitions": [

front/plugins/nmap_dev_scan/nmap_dev.py

@@ -39,7 +39,6 @@ LOG_FILE = os.path.join(LOG_PATH, f'script.{pluginName}.log')
 RESULT_FILE = os.path.join(LOG_PATH, f'last_result.{pluginName}.log')
 
 
-
 def main():
 
     mylog('verbose', [f'[{pluginName}] In script'])     
@@ -48,9 +47,10 @@ def main():
     db = DB()  # instance of class DB
     db.open()
 
-
     timeout = get_setting_value('NMAPDEV_RUN_TIMEOUT')
+    fakeMac = get_setting_value('NMAPDEV_FAKE_MAC')
     subnets = get_setting_value('SCAN_SUBNETS')
+    args    = get_setting_value('NMAPDEV_ARGS')
 
     mylog('verbose', [f'[{pluginName}] subnets: ', subnets])    
 
@@ -58,7 +58,7 @@ def main():
     # Initialize the Plugin obj output file
     plugin_objects = Plugin_Objects(RESULT_FILE)
 
-    unique_devices = execute_scan(subnets, timeout)
+    unique_devices = execute_scan(subnets, timeout, fakeMac, args)
 
     mylog('verbose', [f'[{pluginName}] Devices found: {len(unique_devices)}'])   
 
@@ -85,17 +85,17 @@ def main():
 #===============================================================================
 # Execute scan
 #===============================================================================
-def execute_scan(subnets_list, timeout):
+def execute_scan(subnets_list, timeout, fakeMac, args):
     devices_list = []
 
     for interface in subnets_list:
-        nmap_output = execute_scan_on_interface(interface, timeout)
+        nmap_output = execute_scan_on_interface(interface, timeout, args)
 
         # mylog('verbose', [f"[{pluginName}] nmap_output XML: ", nmap_output])
 
         if nmap_output:  # Proceed only if nmap output is not empty
             # Parse the XML output using python-nmap
-            devices = parse_nmap_xml(nmap_output, interface)
+            devices = parse_nmap_xml(nmap_output, interface, fakeMac)
 
             for device in devices:
                 # Append to devices_list only if both IP and MAC addresses are present
@@ -112,9 +112,9 @@ def execute_scan(subnets_list, timeout):
 
 
 
-def execute_scan_on_interface (interface, timeout):
+def execute_scan_on_interface (interface, timeout, args):
     # Prepare command arguments
-    scan_args = get_setting_value('NMAPDEV_ARGS').split() + interface.replace('--interface=','-e ').split()
+    scan_args = args.split() + interface.replace('--interface=','-e ').split()
 
     mylog('verbose', [f'[{pluginName}] scan_args: ', scan_args])   
     
@@ -128,7 +128,7 @@ def execute_scan_on_interface (interface, timeout):
     return result
 
 
-def parse_nmap_xml(xml_output, interface):
+def parse_nmap_xml(xml_output, interface, fakeMac):
     devices_list = []
 
     try:
@@ -161,7 +161,11 @@ def parse_nmap_xml(xml_output, interface):
             mylog('verbose', [f"[{pluginName}] Hostname: {hostname}, IP: {ip}, MAC: {mac}, Vendor: {vendor}"])
 
             # Only include devices with both IP and MAC addresses
-            if ip != '' and mac != '':
+            if (ip != '' and mac != '') or (ip != '' and fakeMac):
+
+                if mac == '' and fakeMac:
+                    mac = string_to_mac_hash(ip)
+
                 devices_list.append({
                     'name': hostname,
                     'ip': ip,
@@ -171,7 +175,7 @@ def parse_nmap_xml(xml_output, interface):
                 })
             else:
                 # MAC or IP missing
-                mylog('verbose', [f"[{pluginName}] Skipping: {hostname}, IP or MAC missing"])
+                mylog('verbose', [f"[{pluginName}] Skipping: {hostname}, IP or MAC missing, or NMAPDEV_GENERATE_MAC setting not enabled"])
 
 
     except Exception as e:
@@ -180,7 +184,14 @@ def parse_nmap_xml(xml_output, interface):
     return devices_list
           
     
+def string_to_mac_hash(input_string):
+    # Calculate a hash using SHA-256
+    sha256_hash = hashlib.sha256(input_string.encode()).hexdigest()
+
+    # Take the first 12 characters of the hash and format as a MAC address
+    mac_hash = ':'.join(sha256_hash[i:i+2] for i in range(0, 12, 2))
     
+    return mac_hash
 
 #===============================================================================
 # BEGIN

server/__main__.py

@@ -105,7 +105,7 @@ def main ():
         pluginsState = check_and_run_user_event(db, all_plugins, pluginsState)
 
         # Update API endpoints              
-        update_api(db, all_plugins)
+        update_api(db, all_plugins, False)
 
         # proceed if 1 minute passed
         if conf.last_scan_run + datetime.timedelta(minutes=1) < conf.loop_start_time :

server/api.py

@@ -24,7 +24,7 @@ stop_event = threading.Event()  # Event to signal thread termination
 #===============================================================================
 # API
 #===============================================================================
-def update_api(db, all_plugins, updateOnlyDataSources=[], is_ad_hoc_user_event=False):
+def update_api(db, all_plugins, forceUpdate, updateOnlyDataSources=[], is_ad_hoc_user_event=False):
     mylog('debug', ['[API] Update API starting'])
 
     # Start periodic write if not running
@@ -57,7 +57,7 @@ def update_api(db, all_plugins, updateOnlyDataSources=[], is_ad_hoc_user_event=F
     # Save selected database tables
     for dsSQL in dataSourcesSQLs:
         if not updateOnlyDataSources or dsSQL[0] in updateOnlyDataSources:
-            api_endpoint_class(db, dsSQL[1], folder + 'table_' + dsSQL[0] + '.json', is_ad_hoc_user_event)
+            api_endpoint_class(db, forceUpdate, dsSQL[1], folder + 'table_' + dsSQL[0] + '.json', is_ad_hoc_user_event)
             
     # Start the GraphQL server
     graphql_port_value = get_setting_value("GRAPHQL_PORT")
@@ -76,7 +76,7 @@ def update_api(db, all_plugins, updateOnlyDataSources=[], is_ad_hoc_user_event=F
 
 #-------------------------------------------------------------------------------
 class api_endpoint_class:
-    def __init__(self, db, query, path, is_ad_hoc_user_event=False):        
+    def __init__(self, db, forceUpdate, query, path, is_ad_hoc_user_event=False):        
         global apiEndpoints
 
         current_time = timeNowTZ()
@@ -125,17 +125,17 @@ class api_endpoint_class:
             apiEndpoints.append(self)
 
         # Needs to be called for initial updates
-        self.try_write()
+        self.try_write(forceUpdate)
 
     #----------------------------------------
-    def try_write(self):
+    def try_write(self, forceUpdate):
         current_time = timeNowTZ()
 
         # Debugging info to understand the issue 
         # mylog('debug', [f'[API] api_endpoint_class: {self.fileName} is_ad_hoc_user_event {self.is_ad_hoc_user_event} last_update_time={self.last_update_time}, debounce time={self.last_update_time + datetime.timedelta(seconds=self.debounce_interval)}.'])
 
         # Only attempt to write if the debounce time has passed
-        if self.needsUpdate and (self.changeDetectedWhen is None or current_time > (self.changeDetectedWhen + datetime.timedelta(seconds=self.debounce_interval))):
+        if forceUpdate == True or (self.needsUpdate and (self.changeDetectedWhen is None or current_time > (self.changeDetectedWhen + datetime.timedelta(seconds=self.debounce_interval)))):
 
             mylog('debug', [f'[API] api_endpoint_class: Writing {self.fileName} after debounce.'])
 

server/initialise.py

@@ -332,7 +332,7 @@ def importConfigs (db, all_plugins):
     
     # setup execution schedules AFTER OVERRIDE handling
 
-    mylog('verbose', [f"[Config] c_d {c_d}"])
+    # mylog('verbose', [f"[Config] c_d {c_d}"])
 
     for plugin in all_plugins:        
         # Setup schedules
@@ -383,7 +383,7 @@ def importConfigs (db, all_plugins):
     db.commitDB()
 
     #  update only the settings datasource
-    update_api(db, all_plugins, ["settings"])  
+    update_api(db, all_plugins, True, ["settings"])  
     
     # run plugins that are modifying the config   
     run_plugin_scripts(db, all_plugins, 'before_config_save' )

server/plugin.py

@@ -467,7 +467,7 @@ def execute_plugin(db, all_plugins, plugin, pluginsState = plugins_state() ):
         pluginsState = process_plugin_events(db, plugin, pluginsState, sqlParams)
 
         # update API endpoints
-        update_api(db, all_plugins, ["plugins_events","plugins_objects", "plugins_history", "appevents"])  
+        update_api(db, all_plugins, False, ["plugins_events","plugins_objects", "plugins_history", "appevents"])  
     
     return pluginsState
 
@@ -873,7 +873,7 @@ def check_and_run_user_event(db, all_plugins, pluginsState):
             execution_log.finalize_event("run")               
         elif event == 'update_api':
             # async handling
-            update_api(db, all_plugins, param.split(','), True)
+            update_api(db, all_plugins, False, param.split(','), True)
             
         else:
             mylog('minimal', ['[check_and_run_user_event] WARNING: Unhandled event in execution queue: ', event, ' | ', param])