diff --git a/reference/crowdsec.md b/reference/crowdsec.md new file mode 100644 index 0000000..de2da47 --- /dev/null +++ b/reference/crowdsec.md @@ -0,0 +1,611 @@ +--- +created: '2026-04-15T18:21:24+00:00' +id: bfb22588-d606-469c-a04a-f1fd82088566 +keywords: +- crowdsec +- cscli +- bouncer +- lapi +- capi +- appsec +- waf +- hub +- scenarios +- parsers +- profiles +- acquisition +- remediation +- traefik bouncer +- iptables +- nftables +- community blocklist +- CTI +links: [] +modified: '2026-04-15T18:21:24+00:00' +summary: Comprehensive CrowdSec IDS/IPS reference covering architecture, configuration, + log processing pipeline, detection scenarios, remediation bouncers, AppSec/WAF, + Hub management, cscli CLI, and API configuration. Derived from official docs at + docs.crowdsec.net. +tags: +- crowdsec +- security +- ids +- ips +- waf +- appsec +- firewall +- devops +- infrastructure +- reference +title: CrowdSec — Intrusion Detection & Prevention System (IDS/IPS) +--- + +--- +id: ref-crowdsec +title: "CrowdSec — Intrusion Detection & Prevention System (IDS/IPS)" +created: 2025-07-11 +modified: 2025-07-11 +tags: + - crowdsec + - security + - ids + - ips + - waf + - appsec + - firewall + - devops + - infrastructure + - reference +links: + - rel: companion + href: reference/traefik.md + title: "Traefik Proxy — Reference Documentation" + - rel: companion + href: reference/docker-compose.md + title: "Docker Compose — Reference Documentation" +--- + +# CrowdSec Documentation Reference + +> CrowdSec is an open-source, collaborative intrusion detection and prevention system (IDS/IPS) that analyzes logs and HTTP requests to detect malicious behavior, block attackers, and share threat intelligence across a community network. +> +> **Official docs**: (accessed 2025) +> **Hub**: +> **GitHub**: + +--- + +## Architecture Overview + +CrowdSec is a modular security stack with these core components: + +| Component | Abbreviation | Role | +|---|---|---| +| **Security Engine** | — | The overall IDS/IPS system, comprising Log Processor + Local API | +| **Log Processor** | LP | Detects bad behaviors by acquiring/parsing logs and matching against Scenarios and AppSec rules | +| **Local API** | LAPI | Middleman between Log Processors, Remediation Components, and the Central API. Stores alerts, creates decisions | +| **Remediation Components** | Bouncers | Enforce decisions — block IPs via iptables, firewalls, web servers, reverse proxies | +| **Central API** | CAPI | Gateway for network participants; aggregates attack signals and redistributes curated Community Blocklists | +| **Console** | — | Web UI for managing alerts, decisions, blocklists, and API keys | + +### Request Flow + +``` +Logs / HTTP Requests + │ + ▼ + ┌─────────────┐ + │ Log Processor│──── Scenarios matched ────▶ Alerts + │ (detection) │──── AppSec rules matched ──▶ Alerts + └──────┬───────┘ + │ alerts + ▼ + ┌─────────────┐ ┌──────────────┐ + │ Local API │◄────────►│ Central API │ + │ (decisions) │ │ (community) │ + └──────┬───────┘ └──────────────┘ + │ decisions + ▼ + ┌─────────────┐ + │ Bouncers │─── iptables / nftables / nginx / traefik / etc. + │ (enforcement)│ + └─────────────┘ +``` + +### Deployment Options + +- **Single machine**: Security Engine + bouncer on one host +- **Distributed**: Multiple Log Processors report to a central LAPI; bouncers query LAPI +- **Container/Kubernetes**: Docker data source or Helm chart available +- **Log-pit setup**: Run CrowdSec next to existing log aggregators (rsyslog, Loki) + +--- + +## Configuration + +### Main Config: `/etc/crowdsec/config.yaml` + +Full YAML configuration with these top-level sections: + +#### `common` — Daemon settings + +```yaml +common: + daemonize: true|false + pid_dir: /var/run/ + log_media: file|stdout|syslog + log_level: error|info|debug|trace + log_dir: /var/log/ + working_dir: /var/lib/crowdsec/ + log_max_size: 500 # MB + log_max_age: 28 # days + log_max_files: 3 + compress_logs: true|false + log_format: text|json +``` + +#### `config_paths` — Directory layout + +```yaml +config_paths: + config_dir: /etc/crowdsec/ + data_dir: /var/lib/crowdsec/data/ + simulation_path: /etc/crowdsec/simulation.yaml + hub_dir: /etc/crowdsec/hub/ + index_path: /etc/crowdsec/hub/.index.json + notification_dir: /etc/crowdsec/notifications/ + plugin_dir: /var/lib/crowdsec/plugins/ +``` + +#### `crowdsec_service` — Agent (Log Processor) settings + +```yaml +crowdsec_service: + enable: true + acquisition_path: /etc/crowdsec/acquis.yaml + acquisition_dir: /etc/crowdsec/acquis.d/ + parser_routines: 1 + buckets_routines: 1 + output_routines: 1 +``` + +#### `db_config` — Database (SQLite, MySQL, PostgreSQL) + +```yaml +db_config: + type: sqlite|mysql|postgresql|pgx + db_path: /var/lib/crowdsec/data/crowdsec.db # sqlite: file path; mysql/pgx: socket + use_wal: true # sqlite WAL mode (recommended) + user: "" # mysql/pg only + password: "" # mysql/pg only + db_name: "" # mysql/pg only + host: "" # mysql/pg only (omit for socket) + port: 3306 # mysql/pg only + sslmode: require|disable # pg only + max_open_conns: 100 + decision_bulk_size: 1000 + flush: + max_items: 5000 + max_age: 168h # 7 days + metrics_max_age: 2h + bouncers_autodelete: + cert: 720h + api_key: 720h + agents_autodelete: + cert: 720h + login_password: 720h +``` + +#### `api` — Local API + Central API + CTI + +```yaml +api: + server: + enable: true + listen_uri: 127.0.0.1:8080 + profiles_path: /etc/crowdsec/profiles.yaml + use_forwarded_for_headers: false + trusted_ips: + - 127.0.0.1 + online_client: + sharing: true|false # push signals to CAPI + pull: + community: true|false # pull community blocklist + blocklists: true|false # pull CrowdSec blocklists + credentials_path: /etc/crowdsec/online_api_credentials.yaml + tls: + cert_file: "" + key_file: "" + client_verification: NoClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert + ca_cert_path: "" + auto_registration: + enabled: false + token: "" + allowed_ranges: [] + client: + credentials_path: /etc/crowdsec/local_api_credentials.yaml + cti: + enabled: true|false + key: "" + cache_timeout: 60m + cache_size: 50 +``` + +#### `prometheus` — Metrics endpoint + +```yaml +prometheus: + enabled: false + level: full|aggregated + listen_addr: 127.0.0.1 + listen_port: 6060 +``` + +### Config Override with `.yaml.local` + +For files that support `.yaml.local` (config.yaml, bouncer configs, simulation.yaml), values in the `.local` file take precedence. Mappings are merged; sequences are replaced. `profiles.yaml` files are read as a whole (concatenated), not merged. + +**Environment variables** can be used in config values: `${DB_PASSWORD}`. Undefined variables are left as literal strings (v1.5.0+). + +### Key Configuration Directories + +| Path | Purpose | +|---|---| +| `/etc/crowdsec/config.yaml` | Main configuration | +| `/etc/crowdsec/acquis.d/*.yaml` | Log source / datasource definitions | +| `/etc/crowdsec/bouncers/*.yaml` | Bouncer (remediation) configs | +| `/etc/crowdsec/parsers/s00-raw/` | Raw format parsers (syslog, journal, etc.) | +| `/etc/crowdsec/parsers/s01-parse/` | Service-specific parsers (nginx, ssh, apache) | +| `/etc/crowdsec/parsers/s02-enrich/` | Enrichment parsers and whitelists | +| `/etc/crowdsec/scenarios/` | Detection scenarios | +| `/etc/crowdsec/profiles.yaml` | Decision profiles (duration, type, scope, notifications) | +| `/etc/crowdsec/notifications/*.yaml` | Notification plugin configs (slack, email, splunk) | +| `/etc/crowdsec/appsec-configs/*.yaml` | WAF configuration (which rules, inband/outofband, hooks) | +| `/etc/crowdsec/appsec-rules/*.yaml` | Individual WAF rules | +| `/etc/crowdsec/collections/*.yaml` | Installed collections | +| `/etc/crowdsec/contexts/*.yaml` | Alert context data sent to LAPI | +| `/etc/crowdsec/hub/` | Local hub mirror (do not modify) | + +--- + +## Log Processing Pipeline + +### Parser Stages + +Parsers run in three ordered stages, processing events sequentially: + +1. **s00-raw**: Base format parsers — syslog, journald, docker, cloudwatch, etc. +2. **s01-parse**: Service-specific parsers — extract structured fields from nginx, ssh, apache, etc. +3. **s02-enrich**: Enrichment — geoip, whitelists, additional metadata + +Each parser has: + +- `filter`: Expression to decide if event enters the parser +- `onsuccess`: What to do on success (`next_stage`, `next_parser`, `discard`) +- `grok`: Pattern matching with `pattern_syntax` definitions +- `statics`: Set fields via literal values or expressions (`expression:`) +- `meta` vs `parsed`: Meta fields are used for routing/matching; parsed fields store extracted data + +### Parser Example + +```yaml +onsuccess: next_stage +filter: "evt.Parsed.program == 'myservice'" +name: crowdsecurity/myservice-logs +description: "Parse myservice logs" +pattern_syntax: + MYSERVICE_BADPASSWORD: "bad password for user '%{USERNAME:user}' from '%{IP:source_ip}'" +nodes: + - grok: + name: MYSERVICE_BADPASSWORD + apply_on: message + statics: + - meta: log_type + value: myservice_failed_auth +statics: + - meta: service + value: myservice + - meta: source_ip + expression: "evt.Parsed.source_ip" +``` + +### Testing Parsers + +```bash +cscli hubtest create myservice-logs --type syslog +# Edit .tests/myservice-logs/config.yaml, .tests/myservice-logs/myservice-logs.log +cscli hubtest run myservice-logs +cscli hubtest explain myservice-logs +``` + +--- + +## Scenarios (Detection Rules) + +Scenarios define behavioral detection patterns using "buckets" that accumulate events and overflow when thresholds are met. + +### Scenario Types + +- **leaky**: Bucket drains over time (`leakspeed`). Events add to bucket; if it reaches `capacity` before draining, it overflows → alert +- **trigger**: Single event match → immediate alert + +### Scenario Example (Leaky — Brute Force) + +```yaml +type: leaky +name: crowdsecurity/myservice-bf +description: "Detect myservice bruteforce" +filter: "evt.Meta.log_type == 'myservice_failed_auth'" +leakspeed: "10s" +capacity: 5 +groupby: evt.Meta.source_ip +blackhole: 1m +reprocess: true +labels: + service: myservice + confidence: 3 + spoofable: 0 + classification: + - attack.T1110 + behavior: "myservice:bruteforce" + label: "myservice bruteforce" + service: myservice + remediation: true +``` + +### Key Scenario Fields + +| Field | Description | +|---|---| +| `type` | `leaky` or `trigger` | +| `filter` | Expression to match events into this bucket | +| `leakspeed` | Rate at which bucket drains (e.g., `"10s"` = 1 event per 10s) | +| `capacity` | Events needed to overflow | +| `groupby` | Field to partition buckets (usually `source_ip`) | +| `blackhole` | Duration to suppress re-triggering for same group | +| `reprocess` | Whether to ingest the alert for further processing | +| `labels.remediation` | `true` to enable bouncer blocking for this scenario | + +--- + +## Profiles (Decision Configuration) + +**Path**: `/etc/crowdsec/profiles.yaml` + +Profiles control what decisions are generated when a scenario triggers an alert: + +- **duration**: How long the remediation lasts (default: `4h`) +- **type**: Remediation type — `ban`, `captcha`, etc. +- **scope**: What gets blocked — `ip`, `range`, `session`, etc. +- **notifications**: Enable notification plugins (slack, email, splunk, etc.) +- **debug**: Enable per-profile debugging + +```yaml +name: default_ip_remediation +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +on_success: break +``` + +Profiles are evaluated in order; first matching profile wins (unless `on_success: next`). + +--- + +## Remediation Components (Bouncers) + +Bouncers enforce decisions by querying the Local API for active decisions and applying them at the infrastructure level. + +### Available Bouncers (Hub) + +- **iptables** / **nftables**: Linux firewall blocking +- **nginx** / **openresty**: Web server / reverse proxy blocking +- **traefik**: Traefik reverse proxy blocking +- **cs-firewall-bouncer**: Generic firewall (iptables/nftables) +- **cs-windows-firewall-bouncer**: Windows Firewall +- **cloudflare**: Cloudflare WAF integration +- **blocklist-mirror**: Expose decisions as a blocklist for other tools + +### Bouncer Registration + +```bash +# Add a bouncer (generates API key) +sudo cscli bouncers add mybouncer +# List registered bouncers +sudo cscli bouncers list +# Delete a bouncer +sudo cscli bouncers delete mybouncer +``` + +The generated API key goes into the bouncer's YAML config file (e.g., `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`). + +### Authentication to LAPI + +- **Bouncers**: API key auth — read-only, can only get decisions +- **Machines**: Login/password — can push alerts and manage decisions +- **TLS client certs**: Alternative auth; requires CA cert and allowed OUs configured in LAPI + +--- + +## AppSec / WAF Component + +Since v1.6.0, CrowdSec includes a built-in Web Application Firewall (AppSec Component) that analyzes HTTP requests in real-time. + +### Setup (Nginx/OpenResty) + +1. **Install collections**: + + ```bash + sudo cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules + ``` + +2. **Configure acquisition** (`/etc/crowdsec/acquis.d/appsec.yaml`): + + ```yaml + appsec_configs: + - crowdsecurity/appsec-default + labels: + type: appsec + listen_addr: 127.0.0.1:7422 + source: appsec + ``` + +3. **Configure bouncer** — add to bouncer config: + + ``` + APPSEC_URL=http://127.0.0.1:7422 + ``` + +4. **Restart services**: + + ```bash + sudo systemctl restart crowdsec + sudo systemctl restart nginx + ``` + +### AppSec Request Flow + +1. Incoming HTTP request hits Nginx/Traefik +2. Remediation Component forwards request to AppSec (port 7422) +3. AppSec evaluates against loaded rules +4. Response: `{"action":"allow","http_status":200}` or `{"action":"ban","http_status":403}` +5. Web server enforces the decision + +### Monitoring + +```bash +cscli metrics show appsec +``` + +--- + +## Hub (Community Configuration Repository) + +The CrowdSec Hub at hosts ready-to-use configurations maintained by CrowdSec team and the community: + +| Type | Purpose | +|---|---| +| **Collections** | Bundles of parsers + scenarios for specific services (sshd, nginx, apache, docker, etc.) | +| **Parsers** | Individual log parsers for specific services | +| **Scenarios** | Individual detection rules | +| **Postoverflows** | Enrichment and whitelist rules | +| **AppSec Configs** | WAF rule configurations | +| **AppSec Rules** | Individual WAF detection rules | +| **Remediation Components** | Bouncer integrations | + +### Hub Management + +```bash +# Install a collection +sudo cscli collections install crowdsecurity/sshd + +# List installed +sudo cscli collections list +sudo cscli parsers list +sudo cscli scenarios list + +# Update all +sudo cscli hub upgrade + +# Inspect a collection's contents +sudo cscli collections inspect crowdsecurity/sshd +``` + +--- + +## cscli — Command-Line Reference + +The `cscli` tool is the primary management interface for CrowdSec. + +### Key Commands + +| Command | Purpose | +|---|---| +| `cscli alerts list` | View active alerts | +| `cscli decisions list` | View active blocking decisions | +| `cscli decisions delete --range` | Delete decisions by IP/CIDR | +| `cscli bouncers list/add/delete` | Manage remediation components | +| `cscli machines list/add/validate/delete` | Manage LAPI machines | +| `cscli lapi register -u ` | Register to a remote LAPI | +| `cscli collections install/upgrade/inspect/list` | Manage hub collections | +| `cscli parsers list/install` | Manage parsers | +| `cscli scenarios list/simulation` | Manage scenarios | +| `cscli hub upgrade` | Update all hub items | +| `cscli metrics show` | View Prometheus metrics (parsers, buckets, appsec) | +| `cscli metrics show bouncers` | Bouncer-specific metrics | +| `cscli metrics show appsec` | WAF metrics | +| `cscli explain` | Explain log processing pipeline | +| `cscli simulation` | Toggle simulation mode (detect but don't block) | +| `cscli allowlists` | Manage centralized allowlists | +| `cscli capi status` | Check Central API connectivity | +| `cscli console enroll` | Enroll in the CrowdSec Console | +| `cscli hubtest create/run/explain` | Test custom parsers/scenarios | +| `cscli config show` | Display current configuration | +| `cscli support` | Diagnostic tools | +| `cscli version` | Display CrowdSec version | + +### Simulation Mode + +```bash +# Put specific scenario in simulation (detect but don't block) +sudo cscli simulation enable crowdsecurity/ssh-bf +# Put ALL scenarios in simulation +sudo cscli simulation enable --all +# List simulation status +sudo cscli simulation status +``` + +--- + +## Local API (LAPI) Management + +### Setup + +- Default listen: `127.0.0.1:8080` +- Swagger docs available at the `/api` endpoint +- Two authentication types: bouncers (API key, read-only) and machines (login/password, read-write) + +### Remote LAPI Registration + +```bash +# On agent machine: +sudo cscli lapi register -u http://: +# On LAPI server — validate: +sudo cscli machines validate +sudo cscli machines list +``` + +### TLS Configuration + +- Cert/key files for server TLS +- Client cert verification modes +- Allowed OUs for agents and bouncers +- CRL (Certificate Revocation List) support + +--- + +## Central API (CAPI) & Community Blocklists + +- Participants share attack signals and receive curated Community Blocklists +- Sharing more data → receiving more blocklist entries (plan-dependent) +- Configuration in `api.server.online_client` +- Pull controls: `community` (crowd-sourced) and `blocklists` (curated feeds) + +--- + +## CTI (Cyber Threat Intelligence) API + +- REST API for IP reputation lookups +- IP reputation scores, attack context +- SIEM integration support +- Configured in `api.cti` section +- Caching with configurable timeout and size + +--- + +## Related References in Knowledge Base + +- **Traefik**: `reference/traefik.md` — CrowdSec has a native Traefik bouncer +- **Docker Compose**: `reference/docker-compose.md` — CrowdSec supports Docker as a log datasource and can run as a container \ No newline at end of file