vh/VuongKB

Fork 0

Files

T

Atlas 44f6b053de Create: reference/crowdsec.md

2026-04-15 11:21:24 -07:00

18 KiB

Raw Blame History

created, id, keywords, links, modified, summary, tags, title

created

keywords

links

modified

summary

CrowdSec Documentation Reference

CrowdSec is an open-source, collaborative intrusion detection and prevention system (IDS/IPS) that analyzes logs and HTTP requests to detect malicious behavior, block attackers, and share threat intelligence across a community network.

Official docs: https://docs.crowdsec.net/ (accessed 2025) Hub: https://hub.crowdsec.net/ GitHub: https://github.com/crowdsecurity/crowdsec

Architecture Overview

CrowdSec is a modular security stack with these core components:

Component	Abbreviation	Role
Security Engine	—	The overall IDS/IPS system, comprising Log Processor + Local API
Log Processor	LP	Detects bad behaviors by acquiring/parsing logs and matching against Scenarios and AppSec rules
Local API	LAPI	Middleman between Log Processors, Remediation Components, and the Central API. Stores alerts, creates decisions
Remediation Components	Bouncers	Enforce decisions — block IPs via iptables, firewalls, web servers, reverse proxies
Central API	CAPI	Gateway for network participants; aggregates attack signals and redistributes curated Community Blocklists
Console	—	Web UI for managing alerts, decisions, blocklists, and API keys

Request Flow

Logs / HTTP Requests
        │
        ▼
  ┌─────────────┐
  │ Log Processor│──── Scenarios matched ────▶ Alerts
  │ (detection)  │──── AppSec rules matched ──▶ Alerts
  └──────┬───────┘
         │ alerts
         ▼
  ┌─────────────┐          ┌──────────────┐
  │  Local API  │◄────────►│  Central API │
  │ (decisions) │          │ (community)  │
  └──────┬───────┘          └──────────────┘
         │ decisions
         ▼
  ┌─────────────┐
  │  Bouncers   │─── iptables / nftables / nginx / traefik / etc.
  │ (enforcement)│
  └─────────────┘

Deployment Options

Single machine: Security Engine + bouncer on one host
Distributed: Multiple Log Processors report to a central LAPI; bouncers query LAPI
Container/Kubernetes: Docker data source or Helm chart available
Log-pit setup: Run CrowdSec next to existing log aggregators (rsyslog, Loki)

Configuration

Main Config: `/etc/crowdsec/config.yaml`

Full YAML configuration with these top-level sections:

`common` — Daemon settings

common:
  daemonize: true|false
  pid_dir: /var/run/
  log_media: file|stdout|syslog
  log_level: error|info|debug|trace
  log_dir: /var/log/
  working_dir: /var/lib/crowdsec/
  log_max_size: 500        # MB
  log_max_age: 28          # days
  log_max_files: 3
  compress_logs: true|false
  log_format: text|json

`config_paths` — Directory layout

config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /var/lib/crowdsec/plugins/

`crowdsec_service` — Agent (Log Processor) settings

crowdsec_service:
  enable: true
  acquisition_path: /etc/crowdsec/acquis.yaml
  acquisition_dir: /etc/crowdsec/acquis.d/
  parser_routines: 1
  buckets_routines: 1
  output_routines: 1

`db_config` — Database (SQLite, MySQL, PostgreSQL)

db_config:
  type: sqlite|mysql|postgresql|pgx
  db_path: /var/lib/crowdsec/data/crowdsec.db  # sqlite: file path; mysql/pgx: socket
  use_wal: true                                    # sqlite WAL mode (recommended)
  user: ""                                         # mysql/pg only
  password: ""                                     # mysql/pg only
  db_name: ""                                      # mysql/pg only
  host: ""                                         # mysql/pg only (omit for socket)
  port: 3306                                       # mysql/pg only
  sslmode: require|disable                         # pg only
  max_open_conns: 100
  decision_bulk_size: 1000
  flush:
    max_items: 5000
    max_age: 168h                    # 7 days
    metrics_max_age: 2h
    bouncers_autodelete:
      cert: 720h
      api_key: 720h
    agents_autodelete:
      cert: 720h
      login_password: 720h

`api` — Local API + Central API + CTI

api:
  server:
    enable: true
    listen_uri: 127.0.0.1:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    use_forwarded_for_headers: false
    trusted_ips:
      - 127.0.0.1
    online_client:
      sharing: true|false       # push signals to CAPI
      pull:
        community: true|false   # pull community blocklist
        blocklists: true|false  # pull CrowdSec blocklists
      credentials_path: /etc/crowdsec/online_api_credentials.yaml
    tls:
      cert_file: ""
      key_file: ""
      client_verification: NoClientCert|VerifyClientCertIfGiven|RequireAndVerifyClientCert
      ca_cert_path: ""
    auto_registration:
      enabled: false
      token: ""
      allowed_ranges: []
  client:
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  cti:
    enabled: true|false
    key: ""
    cache_timeout: 60m
    cache_size: 50

`prometheus` — Metrics endpoint

prometheus:
  enabled: false
  level: full|aggregated
  listen_addr: 127.0.0.1
  listen_port: 6060

Config Override with `.yaml.local`

For files that support .yaml.local (config.yaml, bouncer configs, simulation.yaml), values in the .local file take precedence. Mappings are merged; sequences are replaced. profiles.yaml files are read as a whole (concatenated), not merged.

Environment variables can be used in config values: ${DB_PASSWORD}. Undefined variables are left as literal strings (v1.5.0+).

Key Configuration Directories

Path	Purpose
`/etc/crowdsec/config.yaml`	Main configuration
`/etc/crowdsec/acquis.d/*.yaml`	Log source / datasource definitions
`/etc/crowdsec/bouncers/*.yaml`	Bouncer (remediation) configs
`/etc/crowdsec/parsers/s00-raw/`	Raw format parsers (syslog, journal, etc.)
`/etc/crowdsec/parsers/s01-parse/`	Service-specific parsers (nginx, ssh, apache)
`/etc/crowdsec/parsers/s02-enrich/`	Enrichment parsers and whitelists
`/etc/crowdsec/scenarios/`	Detection scenarios
`/etc/crowdsec/profiles.yaml`	Decision profiles (duration, type, scope, notifications)
`/etc/crowdsec/notifications/*.yaml`	Notification plugin configs (slack, email, splunk)
`/etc/crowdsec/appsec-configs/*.yaml`	WAF configuration (which rules, inband/outofband, hooks)
`/etc/crowdsec/appsec-rules/*.yaml`	Individual WAF rules
`/etc/crowdsec/collections/*.yaml`	Installed collections
`/etc/crowdsec/contexts/*.yaml`	Alert context data sent to LAPI
`/etc/crowdsec/hub/`	Local hub mirror (do not modify)

Log Processing Pipeline

Parser Stages

Parsers run in three ordered stages, processing events sequentially:

s00-raw: Base format parsers — syslog, journald, docker, cloudwatch, etc.
s01-parse: Service-specific parsers — extract structured fields from nginx, ssh, apache, etc.
s02-enrich: Enrichment — geoip, whitelists, additional metadata

Each parser has:

filter: Expression to decide if event enters the parser
onsuccess: What to do on success (next_stage, next_parser, discard)
grok: Pattern matching with pattern_syntax definitions
statics: Set fields via literal values or expressions (expression:)
meta vs parsed: Meta fields are used for routing/matching; parsed fields store extracted data

Parser Example

onsuccess: next_stage
filter: "evt.Parsed.program == 'myservice'"
name: crowdsecurity/myservice-logs
description: "Parse myservice logs"
pattern_syntax:
  MYSERVICE_BADPASSWORD: "bad password for user '%{USERNAME:user}' from '%{IP:source_ip}'"
nodes:
  - grok:
      name: MYSERVICE_BADPASSWORD
      apply_on: message
      statics:
        - meta: log_type
          value: myservice_failed_auth
statics:
  - meta: service
    value: myservice
  - meta: source_ip
    expression: "evt.Parsed.source_ip"

Testing Parsers

cscli hubtest create myservice-logs --type syslog
# Edit .tests/myservice-logs/config.yaml, .tests/myservice-logs/myservice-logs.log
cscli hubtest run myservice-logs
cscli hubtest explain myservice-logs

Scenarios (Detection Rules)

Scenarios define behavioral detection patterns using "buckets" that accumulate events and overflow when thresholds are met.

Scenario Types

leaky: Bucket drains over time (leakspeed). Events add to bucket; if it reaches capacity before draining, it overflows → alert
trigger: Single event match → immediate alert

Scenario Example (Leaky — Brute Force)

type: leaky
name: crowdsecurity/myservice-bf
description: "Detect myservice bruteforce"
filter: "evt.Meta.log_type == 'myservice_failed_auth'"
leakspeed: "10s"
capacity: 5
groupby: evt.Meta.source_ip
blackhole: 1m
reprocess: true
labels:
  service: myservice
  confidence: 3
  spoofable: 0
  classification:
    - attack.T1110
  behavior: "myservice:bruteforce"
  label: "myservice bruteforce"
  service: myservice
  remediation: true

Key Scenario Fields

Field	Description
`type`	`leaky` or `trigger`
`filter`	Expression to match events into this bucket
`leakspeed`	Rate at which bucket drains (e.g., `"10s"` = 1 event per 10s)
`capacity`	Events needed to overflow
`groupby`	Field to partition buckets (usually `source_ip`)
`blackhole`	Duration to suppress re-triggering for same group
`reprocess`	Whether to ingest the alert for further processing
`labels.remediation`	`true` to enable bouncer blocking for this scenario

Profiles (Decision Configuration)

Path: /etc/crowdsec/profiles.yaml

Profiles control what decisions are generated when a scenario triggers an alert:

duration: How long the remediation lasts (default: 4h)
type: Remediation type — ban, captcha, etc.
scope: What gets blocked — ip, range, session, etc.
notifications: Enable notification plugins (slack, email, splunk, etc.)
debug: Enable per-profile debugging

name: default_ip_remediation
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
  - type: ban
    duration: 4h
on_success: break

Profiles are evaluated in order; first matching profile wins (unless on_success: next).

Remediation Components (Bouncers)

Bouncers enforce decisions by querying the Local API for active decisions and applying them at the infrastructure level.

Available Bouncers (Hub)

iptables / nftables: Linux firewall blocking
nginx / openresty: Web server / reverse proxy blocking
traefik: Traefik reverse proxy blocking
cs-firewall-bouncer: Generic firewall (iptables/nftables)
cs-windows-firewall-bouncer: Windows Firewall
cloudflare: Cloudflare WAF integration
blocklist-mirror: Expose decisions as a blocklist for other tools

Bouncer Registration

# Add a bouncer (generates API key)
sudo cscli bouncers add mybouncer
# List registered bouncers
sudo cscli bouncers list
# Delete a bouncer
sudo cscli bouncers delete mybouncer

The generated API key goes into the bouncer's YAML config file (e.g., /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml).

Authentication to LAPI

Bouncers: API key auth — read-only, can only get decisions
Machines: Login/password — can push alerts and manage decisions
TLS client certs: Alternative auth; requires CA cert and allowed OUs configured in LAPI

AppSec / WAF Component

Since v1.6.0, CrowdSec includes a built-in Web Application Firewall (AppSec Component) that analyzes HTTP requests in real-time.

Setup (Nginx/OpenResty)

Install collections:

sudo cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules

Configure acquisition (/etc/crowdsec/acquis.d/appsec.yaml):

appsec_configs:
  - crowdsecurity/appsec-default
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec

Configure bouncer — add to bouncer config:
```
APPSEC_URL=http://127.0.0.1:7422
```

Restart services:

sudo systemctl restart crowdsec
sudo systemctl restart nginx

AppSec Request Flow

Incoming HTTP request hits Nginx/Traefik
Remediation Component forwards request to AppSec (port 7422)
AppSec evaluates against loaded rules
Response: {"action":"allow","http_status":200} or {"action":"ban","http_status":403}
Web server enforces the decision

Monitoring

cscli metrics show appsec

Hub (Community Configuration Repository)

The CrowdSec Hub at https://hub.crowdsec.net/ hosts ready-to-use configurations maintained by CrowdSec team and the community:

Type	Purpose
Collections	Bundles of parsers + scenarios for specific services (sshd, nginx, apache, docker, etc.)
Parsers	Individual log parsers for specific services
Scenarios	Individual detection rules
Postoverflows	Enrichment and whitelist rules
AppSec Configs	WAF rule configurations
AppSec Rules	Individual WAF detection rules
Remediation Components	Bouncer integrations

Hub Management

# Install a collection
sudo cscli collections install crowdsecurity/sshd

# List installed
sudo cscli collections list
sudo cscli parsers list
sudo cscli scenarios list

# Update all
sudo cscli hub upgrade

# Inspect a collection's contents
sudo cscli collections inspect crowdsecurity/sshd

cscli — Command-Line Reference

The cscli tool is the primary management interface for CrowdSec.

Key Commands

Command	Purpose
`cscli alerts list`	View active alerts
`cscli decisions list`	View active blocking decisions
`cscli decisions delete --range`	Delete decisions by IP/CIDR
`cscli bouncers list/add/delete`	Manage remediation components
`cscli machines list/add/validate/delete`	Manage LAPI machines
`cscli lapi register -u <url>`	Register to a remote LAPI
`cscli collections install/upgrade/inspect/list`	Manage hub collections
`cscli parsers list/install`	Manage parsers
`cscli scenarios list/simulation`	Manage scenarios
`cscli hub upgrade`	Update all hub items
`cscli metrics show`	View Prometheus metrics (parsers, buckets, appsec)
`cscli metrics show bouncers`	Bouncer-specific metrics
`cscli metrics show appsec`	WAF metrics
`cscli explain`	Explain log processing pipeline
`cscli simulation`	Toggle simulation mode (detect but don't block)
`cscli allowlists`	Manage centralized allowlists
`cscli capi status`	Check Central API connectivity
`cscli console enroll`	Enroll in the CrowdSec Console
`cscli hubtest create/run/explain`	Test custom parsers/scenarios
`cscli config show`	Display current configuration
`cscli support`	Diagnostic tools
`cscli version`	Display CrowdSec version

Simulation Mode

# Put specific scenario in simulation (detect but don't block)
sudo cscli simulation enable crowdsecurity/ssh-bf
# Put ALL scenarios in simulation
sudo cscli simulation enable --all
# List simulation status
sudo cscli simulation status

Local API (LAPI) Management

Setup

Default listen: 127.0.0.1:8080
Swagger docs available at the /api endpoint
Two authentication types: bouncers (API key, read-only) and machines (login/password, read-write)

Remote LAPI Registration

# On agent machine:
sudo cscli lapi register -u http://<remote_api>:<port>
# On LAPI server — validate:
sudo cscli machines validate <machineName>
sudo cscli machines list

TLS Configuration

Cert/key files for server TLS
Client cert verification modes
Allowed OUs for agents and bouncers
CRL (Certificate Revocation List) support

Central API (CAPI) & Community Blocklists

Participants share attack signals and receive curated Community Blocklists
Sharing more data → receiving more blocklist entries (plan-dependent)
Configuration in api.server.online_client
Pull controls: community (crowd-sourced) and blocklists (curated feeds)

CTI (Cyber Threat Intelligence) API

REST API for IP reputation lookups
IP reputation scores, attack context
SIEM integration support
Configured in api.cti section
Caching with configurable timeout and size

Traefik: reference/traefik.md — CrowdSec has a native Traefik bouncer
Docker Compose: reference/docker-compose.md — CrowdSec supports Docker as a log datasource and can run as a container

18 KiB Raw Blame History