Create: reference/crowdsec.md

2026-04-15 11:21:24 -07:00
parent da69ead827
commit 44f6b053de
1 changed files with 611 additions and 0 deletions
@@ -0,0 +1,611 @@
 ---
 created: '2026-04-15T18:21:24+00:00'
 id: bfb22588-d606-469c-a04a-f1fd82088566
 keywords:
 - crowdsec
 - cscli
 - bouncer
 - lapi
 - capi
 - appsec
 - waf
 - hub
 - scenarios
 - parsers
 - profiles
 - acquisition
 - remediation
 - traefik bouncer
 - iptables
 - nftables
 - community blocklist
 - CTI
 links: []
 modified: '2026-04-15T18:21:24+00:00'
 summary: Comprehensive CrowdSec IDS/IPS reference covering architecture, configuration,
  log processing pipeline, detection scenarios, remediation bouncers, AppSec/WAF,
  Hub management, cscli CLI, and API configuration. Derived from official docs at
  docs.crowdsec.net.
 tags:
 - crowdsec
 - security
 - ids
 - ips
 - waf
 - appsec
 - firewall
 - devops
 - infrastructure
 - reference
 title: CrowdSec — Intrusion Detection & Prevention System (IDS/IPS)
 ---
 ---
 id: ref-crowdsec
 title: "CrowdSec — Intrusion Detection & Prevention System (IDS/IPS)"
 created: 2025-07-11
 modified: 2025-07-11
 tags:
  - crowdsec
  - security
  - ids
  - ips
  - waf
  - appsec
  - firewall
  - devops
  - infrastructure
  - reference
 links:
  - rel: companion
    href: reference/traefik.md
    title: "Traefik Proxy — Reference Documentation"
  - rel: companion
    href: reference/docker-compose.md
    title: "Docker Compose — Reference Documentation"
 ---
 # CrowdSec Documentation Reference
 > CrowdSec is an open-source, collaborative intrusion detection and prevention system (IDS/IPS) that analyzes logs and HTTP requests to detect malicious behavior, block attackers, and share threat intelligence across a community network.
 >
 > **Official docs**: <https://docs.crowdsec.net/> (accessed 2025)
 > **Hub**: <https://hub.crowdsec.net/>
 > **GitHub**: <https://github.com/crowdsecurity/crowdsec>
 ---
 ## Architecture Overview
 CrowdSec is a modular security stack with these core components:
 | Component | Abbreviation | Role |
 |---|---|---|
 | **Security Engine** | — | The overall IDS/IPS system, comprising Log Processor + Local API |
 | **Log Processor** | LP | Detects bad behaviors by acquiring/parsing logs and matching against Scenarios and AppSec rules |
 | **Local API** | LAPI | Middleman between Log Processors, Remediation Components, and the Central API. Stores alerts, creates decisions |
 | **Remediation Components** | Bouncers | Enforce decisions — block IPs via iptables, firewalls, web servers, reverse proxies |
 | **Central API** | CAPI | Gateway for network participants; aggregates attack signals and redistributes curated Community Blocklists |
 | **Console** | — | Web UI for managing alerts, decisions, blocklists, and API keys |
 ### Request Flow
 ```
 Logs / HTTP Requests
        │
        ▼
  ┌─────────────┐
  │ Log Processor│──── Scenarios matched ────▶ Alerts
  │ (detection)  │──── AppSec rules matched ──▶ Alerts
  └──────┬───────┘
         │ alerts
         ▼
  ┌─────────────┐          ┌──────────────┐
  │  Local API  │◄────────►│  Central API │
  │ (decisions) │          │ (community)  │
  └──────┬───────┘          └──────────────┘
         │ decisions
         ▼
  ┌─────────────┐
  │  Bouncers   │─── iptables / nftables / nginx / traefik / etc.
  │ (enforcement)│
  └─────────────┘
 ```
 ### Deployment Options
 - **Single machine**: Security Engine + bouncer on one host
 - **Distributed**: Multiple Log Processors report to a central LAPI; bouncers query LAPI
 - **Container/Kubernetes**: Docker data source or Helm chart available
 - **Log-pit setup**: Run CrowdSec next to existing log aggregators (rsyslog, Loki)
 ---
 ## Configuration
 ### Main Config: `/etc/crowdsec/config.yaml`
 Full YAML configuration with these top-level sections:
 #### `common` — Daemon settings
 ```yaml
 common:
  daemonize: true|false
  pid_dir: /var/run/
  log_media: file|stdout|syslog
  log_level: error|info|debug|trace
  log_dir: /var/log/
  working_dir: /var/lib/crowdsec/
  log_max_size: 500        # MB
  log_max_age: 28          # days
  log_max_files: 3
  compress_logs: true|false
  log_format: text|json
 ```
 #### `config_paths` — Directory layout
 ```yaml
 config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /var/lib/crowdsec/plugins/
 ```
 #### `crowdsec_service` — Agent (Log Processor) settings
 ```yaml
 crowdsec_service:
  enable: true
  acquisition_path: /etc/crowdsec/acquis.yaml
  acquisition_dir: /etc/crowdsec/acquis.d/
  parser_routines: 1
  buckets_routines: 1
  output_routines: 1
 ```
 #### `db_config` — Database (SQLite, MySQL, PostgreSQL)
 ```yaml
 db_config:
  type: sqlite|mysql|postgresql|pgx
  db_path: /var/lib/crowdsec/data/crowdsec.db  # sqlite: file path; mysql/pgx: socket
  use_wal: true                                    # sqlite WAL mode (recommended)
  user: ""                                         # mysql/pg only
  password: ""                                     # mysql/pg only
  db_name: ""                                      # mysql/pg only
  host: ""                                         # mysql/pg only (omit for socket)
  port: 3306                                       # mysql/pg only
  sslmode: require|disable                         # pg only
  max_open_conns: 100
  decision_bulk_size: 1000
  flush:
    max_items: 5000
    max_age: 168h                    # 7 days
    metrics_max_age: 2h
    bouncers_autodelete:
      cert: 720h
      api_key: 720h
    agents_autodelete:
      cert: 720h
      login_password: 720h
 ```
 #### `api` — Local API + Central API + CTI
 ```yaml
 api:
  server:
    enable: true
    listen_uri: 127.0.0.1:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    use_forwarded_for_headers: false
    trusted_ips:
      - 127.0.0.1
    online_client:
      sharing: true|false       # push signals to CAPI
      pull:
        community: true|false   # pull community blocklist
        blocklists: true|false  # pull CrowdSec blocklists
      credentials_path: /etc/crowdsec/online_api_credentials.yaml
    tls:
      cert_file: ""
      key_file: ""
      client_verification: NoClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert
      ca_cert_path: ""
    auto_registration:
      enabled: false
      token: ""
      allowed_ranges: []
  client:
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  cti:
    enabled: true|false
    key: ""
    cache_timeout: 60m
    cache_size: 50
 ```
 #### `prometheus` — Metrics endpoint
 ```yaml
 prometheus:
  enabled: false
  level: full|aggregated
  listen_addr: 127.0.0.1
  listen_port: 6060
 ```
 ### Config Override with `.yaml.local`
 For files that support `.yaml.local` (config.yaml, bouncer configs, simulation.yaml), values in the `.local` file take precedence. Mappings are merged; sequences are replaced. `profiles.yaml` files are read as a whole (concatenated), not merged.
 **Environment variables** can be used in config values: `${DB_PASSWORD}`. Undefined variables are left as literal strings (v1.5.0+).
 ### Key Configuration Directories
 | Path | Purpose |
 |---|---|
 | `/etc/crowdsec/config.yaml` | Main configuration |
 | `/etc/crowdsec/acquis.d/*.yaml` | Log source / datasource definitions |
 | `/etc/crowdsec/bouncers/*.yaml` | Bouncer (remediation) configs |
 | `/etc/crowdsec/parsers/s00-raw/` | Raw format parsers (syslog, journal, etc.) |
 | `/etc/crowdsec/parsers/s01-parse/` | Service-specific parsers (nginx, ssh, apache) |
 | `/etc/crowdsec/parsers/s02-enrich/` | Enrichment parsers and whitelists |
 | `/etc/crowdsec/scenarios/` | Detection scenarios |
 | `/etc/crowdsec/profiles.yaml` | Decision profiles (duration, type, scope, notifications) |
 | `/etc/crowdsec/notifications/*.yaml` | Notification plugin configs (slack, email, splunk) |
 | `/etc/crowdsec/appsec-configs/*.yaml` | WAF configuration (which rules, inband/outofband, hooks) |
 | `/etc/crowdsec/appsec-rules/*.yaml` | Individual WAF rules |
 | `/etc/crowdsec/collections/*.yaml` | Installed collections |
 | `/etc/crowdsec/contexts/*.yaml` | Alert context data sent to LAPI |
 | `/etc/crowdsec/hub/` | Local hub mirror (do not modify) |
 ---
 ## Log Processing Pipeline
 ### Parser Stages
 Parsers run in three ordered stages, processing events sequentially:
 1. **s00-raw**: Base format parsers — syslog, journald, docker, cloudwatch, etc.
 2. **s01-parse**: Service-specific parsers — extract structured fields from nginx, ssh, apache, etc.
 3. **s02-enrich**: Enrichment — geoip, whitelists, additional metadata
 Each parser has:
 - `filter`: Expression to decide if event enters the parser
 - `onsuccess`: What to do on success (`next_stage`, `next_parser`, `discard`)
 - `grok`: Pattern matching with `pattern_syntax` definitions
 - `statics`: Set fields via literal values or expressions (`expression:`)
 - `meta` vs `parsed`: Meta fields are used for routing/matching; parsed fields store extracted data
 ### Parser Example
 ```yaml
 onsuccess: next_stage
 filter: "evt.Parsed.program == 'myservice'"
 name: crowdsecurity/myservice-logs
 description: "Parse myservice logs"
 pattern_syntax:
  MYSERVICE_BADPASSWORD: "bad password for user '%{USERNAME:user}' from '%{IP:source_ip}'"
 nodes:
  - grok:
      name: MYSERVICE_BADPASSWORD
      apply_on: message
      statics:
        - meta: log_type
          value: myservice_failed_auth
 statics:
  - meta: service
    value: myservice
  - meta: source_ip
    expression: "evt.Parsed.source_ip"
 ```
 ### Testing Parsers
 ```bash
 cscli hubtest create myservice-logs --type syslog
 # Edit .tests/myservice-logs/config.yaml, .tests/myservice-logs/myservice-logs.log
 cscli hubtest run myservice-logs
 cscli hubtest explain myservice-logs
 ```
 ---
 ## Scenarios (Detection Rules)
 Scenarios define behavioral detection patterns using "buckets" that accumulate events and overflow when thresholds are met.
 ### Scenario Types
 - **leaky**: Bucket drains over time (`leakspeed`). Events add to bucket; if it reaches `capacity` before draining, it overflows → alert
 - **trigger**: Single event match → immediate alert
 ### Scenario Example (Leaky — Brute Force)
 ```yaml
 type: leaky
 name: crowdsecurity/myservice-bf
 description: "Detect myservice bruteforce"
 filter: "evt.Meta.log_type == 'myservice_failed_auth'"
 leakspeed: "10s"
 capacity: 5
 groupby: evt.Meta.source_ip
 blackhole: 1m
 reprocess: true
 labels:
  service: myservice
  confidence: 3
  spoofable: 0
  classification:
    - attack.T1110
  behavior: "myservice:bruteforce"
  label: "myservice bruteforce"
  service: myservice
  remediation: true
 ```
 ### Key Scenario Fields
 | Field | Description |
 |---|---|
 | `type` | `leaky` or `trigger` |
 | `filter` | Expression to match events into this bucket |
 | `leakspeed` | Rate at which bucket drains (e.g., `"10s"` = 1 event per 10s) |
 | `capacity` | Events needed to overflow |
 | `groupby` | Field to partition buckets (usually `source_ip`) |
 | `blackhole` | Duration to suppress re-triggering for same group |
 | `reprocess` | Whether to ingest the alert for further processing |
 | `labels.remediation` | `true` to enable bouncer blocking for this scenario |
 ---
 ## Profiles (Decision Configuration)
 **Path**: `/etc/crowdsec/profiles.yaml`
 Profiles control what decisions are generated when a scenario triggers an alert:
 - **duration**: How long the remediation lasts (default: `4h`)
 - **type**: Remediation type — `ban`, `captcha`, etc.
 - **scope**: What gets blocked — `ip`, `range`, `session`, etc.
 - **notifications**: Enable notification plugins (slack, email, splunk, etc.)
 - **debug**: Enable per-profile debugging
 ```yaml
 name: default_ip_remediation
 filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
 decisions:
  - type: ban
    duration: 4h
 on_success: break
 ```
 Profiles are evaluated in order; first matching profile wins (unless `on_success: next`).
 ---
 ## Remediation Components (Bouncers)
 Bouncers enforce decisions by querying the Local API for active decisions and applying them at the infrastructure level.
 ### Available Bouncers (Hub)
 - **iptables** / **nftables**: Linux firewall blocking
 - **nginx** / **openresty**: Web server / reverse proxy blocking
 - **traefik**: Traefik reverse proxy blocking
 - **cs-firewall-bouncer**: Generic firewall (iptables/nftables)
 - **cs-windows-firewall-bouncer**: Windows Firewall
 - **cloudflare**: Cloudflare WAF integration
 - **blocklist-mirror**: Expose decisions as a blocklist for other tools
 ### Bouncer Registration
 ```bash
 # Add a bouncer (generates API key)
 sudo cscli bouncers add mybouncer
 # List registered bouncers
 sudo cscli bouncers list
 # Delete a bouncer
 sudo cscli bouncers delete mybouncer
 ```
 The generated API key goes into the bouncer's YAML config file (e.g., `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`).
 ### Authentication to LAPI
 - **Bouncers**: API key auth — read-only, can only get decisions
 - **Machines**: Login/password — can push alerts and manage decisions
 - **TLS client certs**: Alternative auth; requires CA cert and allowed OUs configured in LAPI
 ---
 ## AppSec / WAF Component
 Since v1.6.0, CrowdSec includes a built-in Web Application Firewall (AppSec Component) that analyzes HTTP requests in real-time.
 ### Setup (Nginx/OpenResty)
 1. **Install collections**:
   ```bash
   sudo cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
   ```
 2. **Configure acquisition** (`/etc/crowdsec/acquis.d/appsec.yaml`):
   ```yaml
   appsec_configs:
     - crowdsecurity/appsec-default
   labels:
     type: appsec
   listen_addr: 127.0.0.1:7422
   source: appsec
   ```
 3. **Configure bouncer** — add to bouncer config:
   ```
   APPSEC_URL=http://127.0.0.1:7422
   ```
 4. **Restart services**:
   ```bash
   sudo systemctl restart crowdsec
   sudo systemctl restart nginx
   ```
 ### AppSec Request Flow
 1. Incoming HTTP request hits Nginx/Traefik
 2. Remediation Component forwards request to AppSec (port 7422)
 3. AppSec evaluates against loaded rules
 4. Response: `{"action":"allow","http_status":200}` or `{"action":"ban","http_status":403}`
 5. Web server enforces the decision
 ### Monitoring
 ```bash
 cscli metrics show appsec
 ```
 ---
 ## Hub (Community Configuration Repository)
 The CrowdSec Hub at <https://hub.crowdsec.net/> hosts ready-to-use configurations maintained by CrowdSec team and the community:
 | Type | Purpose |
 |---|---|
 | **Collections** | Bundles of parsers + scenarios for specific services (sshd, nginx, apache, docker, etc.) |
 | **Parsers** | Individual log parsers for specific services |
 | **Scenarios** | Individual detection rules |
 | **Postoverflows** | Enrichment and whitelist rules |
 | **AppSec Configs** | WAF rule configurations |
 | **AppSec Rules** | Individual WAF detection rules |
 | **Remediation Components** | Bouncer integrations |
 ### Hub Management
 ```bash
 # Install a collection
 sudo cscli collections install crowdsecurity/sshd
 # List installed
 sudo cscli collections list
 sudo cscli parsers list
 sudo cscli scenarios list
 # Update all
 sudo cscli hub upgrade
 # Inspect a collection's contents
 sudo cscli collections inspect crowdsecurity/sshd
 ```
 ---
 ## cscli — Command-Line Reference
 The `cscli` tool is the primary management interface for CrowdSec.
 ### Key Commands
 | Command | Purpose |
 |---|---|
 | `cscli alerts list` | View active alerts |
 | `cscli decisions list` | View active blocking decisions |
 | `cscli decisions delete --range` | Delete decisions by IP/CIDR |
 | `cscli bouncers list/add/delete` | Manage remediation components |
 | `cscli machines list/add/validate/delete` | Manage LAPI machines |
 | `cscli lapi register -u <url>` | Register to a remote LAPI |
 | `cscli collections install/upgrade/inspect/list` | Manage hub collections |
 | `cscli parsers list/install` | Manage parsers |
 | `cscli scenarios list/simulation` | Manage scenarios |
 | `cscli hub upgrade` | Update all hub items |
 | `cscli metrics show` | View Prometheus metrics (parsers, buckets, appsec) |
 | `cscli metrics show bouncers` | Bouncer-specific metrics |
 | `cscli metrics show appsec` | WAF metrics |
 | `cscli explain` | Explain log processing pipeline |
 | `cscli simulation` | Toggle simulation mode (detect but don't block) |
 | `cscli allowlists` | Manage centralized allowlists |
 | `cscli capi status` | Check Central API connectivity |
 | `cscli console enroll` | Enroll in the CrowdSec Console |
 | `cscli hubtest create/run/explain` | Test custom parsers/scenarios |
 | `cscli config show` | Display current configuration |
 | `cscli support` | Diagnostic tools |
 | `cscli version` | Display CrowdSec version |
 ### Simulation Mode
 ```bash
 # Put specific scenario in simulation (detect but don't block)
 sudo cscli simulation enable crowdsecurity/ssh-bf
 # Put ALL scenarios in simulation
 sudo cscli simulation enable --all
 # List simulation status
 sudo cscli simulation status
 ```
 ---
 ## Local API (LAPI) Management
 ### Setup
 - Default listen: `127.0.0.1:8080`
 - Swagger docs available at the `/api` endpoint
 - Two authentication types: bouncers (API key, read-only) and machines (login/password, read-write)
 ### Remote LAPI Registration
 ```bash
 # On agent machine:
 sudo cscli lapi register -u http://<remote_api>:<port>
 # On LAPI server — validate:
 sudo cscli machines validate <machineName>
 sudo cscli machines list
 ```
 ### TLS Configuration
 - Cert/key files for server TLS
 - Client cert verification modes
 - Allowed OUs for agents and bouncers
 - CRL (Certificate Revocation List) support
 ---
 ## Central API (CAPI) & Community Blocklists
 - Participants share attack signals and receive curated Community Blocklists
 - Sharing more data → receiving more blocklist entries (plan-dependent)
 - Configuration in `api.server.online_client`
 - Pull controls: `community` (crowd-sourced) and `blocklists` (curated feeds)
 ---
 ## CTI (Cyber Threat Intelligence) API
 - REST API for IP reputation lookups
 - IP reputation scores, attack context
 - SIEM integration support
 - Configured in `api.cti` section
 - Caching with configurable timeout and size
 ---
 ## Related References in Knowledge Base
 - **Traefik**: `reference/traefik.md` — CrowdSec has a native Traefik bouncer
 - **Docker Compose**: `reference/docker-compose.md` — CrowdSec supports Docker as a log datasource and can run as a container