I have crowdsec running in a container, next to caddy. they share logs so that crowdsec can parse them and make decisions. caddy has a bouncer at the reverse proxy level.

the issue is that anytime theres an update available for crowdsec i get the following error

time="10-02-2023 18:27:18" level=warning msg="Crowdsec is not the latest version. Current version is 'v1.4.5' and the latest stable version is 'v1.4.6'. Please update it!"
time="10-02-2023 18:27:18" level=warning msg="As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.4.6"
time="10-02-2023 18:27:18" level=fatal msg="Failed to get Hub index : failed to download index: bad http code 404 while requesting https://hub-cdn.crowdsec.net/v1.4.5/.index.json"

and then the crowdsec container exits, putting it into a boot loop.

when its in this statea, the caddy bouncer fails to reach it

{"level":"error","ts":1676053796.0650687,"logger":"crowdsec","msg":"Get \"http://crowdsec:8080/v1/decisions/stream?startup=false\": dial tcp: lookup crowdsec on 127.0.0.11:53: no such host","address":"http://crowdsec:8080/"}

and the default behavior appears to be "ignore crowdsec, let the traffic in"

in short, crowdsec pushes an update which silently disables my bouncer & lets in all traffic, until I notice that crowdsec is boot looping.

did I misconfigure something? is this the intended default behavior? Is every update considering a mandatory breaking change? I am confused.

once I update the container things start behaving normally again

But it doesn't feel great knowing that at any moment the bouncer could just silently stop bouncing

Your bouncer should technically fail in a way if it loses connection to crowdsec which in turn will prevent connections from even passing… ?

Like I know If I kill crowdsec my bouncer will fail and everything will fail to pass through to the server.

that aint what mine does

that would be annoying too though

seems unbelievable that crowdsec pushing a new version immediately breaks all crowdsec containers until they update, but that appears to be the case? i must be missing something cuz that would be completely unusable in production

I don't have the issue of the notification of a new update making it stop working? Can you show us how you deployed everything?

if crowdsec isn't running / or reachable, I'd hope the bouncer wouldn't allow connections to flow if it couldn't exactly process..?

Hello @silverneedles
We missed something when releasing 1.4.6, and the hub branch for 1.4.5 was not created automatically.
I've just created it, the container should work next time it starts

I've not had these issues and I'm still on 1.4.5 however should I be running hub update daily or so? I only do it like once every couple of days..?

the issue was only critical for docker, as it tries to contact the hub on startup
If you are running crowdsec outside of a container, only cscli would have been impacted (ie, you would not have been able to install collections/scenarios/parsers), but crowdsec itself would have worked just fine.
Regarding the frequency of hub upgrade, we do ship a cronjob since 1.4.2 that does it daily. You can probably do it a bit less frequently as the hub is not updated daily most of the time, but it doesn't hurt

it depends how the bouncer is configured
If you use a bouncer in stream mode (firewall bouncer for example), it should keep in cache the list of IPs and continue to work (obviously no new ban will work), but a bug could be present in the bouncer preventing it from working (the caddy bouncer is not written by us, and i've never used it, so i can't say if this is the case or not)
If the bouncer is in live mode, as it needs to contact your LAPI for each request, it won't be able to get an answer, and a well-behaved bouncer should let the traffic pass in this case (you don't want to down your website just because LAPI has crashed)

If it's a reverse proxy level bouncer maybe Caddy is different but Traefik bouncer will just have everything be forbidden if something is wrong with crowdsec not being reachable in the case of I had collections that were no longer in there anymore.. so crowdsec failed but for the most part. It works as I assume it should.

gotcha. to be clear, this is the second time this has happened to me

i have been using crowdsec in docker since 1.4.4 and my contain has been broken by both updates. the error messages were identical the same in both cases, besides the version numbers

was the same mistake made during the update from 1.4.4 -> 1.4.5 ? or do you think i have a misconfiguration somewhere

or should i just expect that the crowdsec container will be broken for a few days after each update is pushed

for reference, these are my configurations

# docker-compose.yaml

version: "3.7"

networks:
  caddy-ol:
    driver: overlay
    external: true

volumes:
  caddy-logs:
  crowdsec-db:

services:
  caddy:
    # image: caddy:2.5.1-alpine
    build: .
    container_name: caddy
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy-ol
      - default
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./caddy-data:/data
      - caddy-logs:/var/log/caddy/
    restart: unless-stopped

  crowdsec:
    image: crowdsecurity/crowdsec
    container_name: crowdsec
    volumes:
      - ./acquis.yaml:/etc/crowdsec/acquis.yaml
      - ./crowdsec-conf:/etc/crowdsec/
      - crowdsec-db:/var/lib/crowdsec/data/
      - caddy-logs:/var/log/caddy/
    environment:
      COLLECTIONS: "crowdsecurity/caddy"
    restart: unless-stopped
    networks:
      - default

and

# acquis.yaml

filenames:
 - /var/log/caddy/*.log
labels:
  type: caddy

tbh i haven't really done much configuring, I installed the caddy collection and made the caddy logs available and everything else is default

good to know that the caddy bouncer is not maintained by crowdsec, but im more concerned with the main crowdsec container becoming unreachable when updates are pushed than I am with the default behavior of the bouncer when it can't reach crowdsec

that definitely shouldn't happen
we recently changed the release process, and we had a few hiccups
We are putting more checks in place to make sure it does not happen again

Odd I've not had this behavior

So what was changed to fix the behavior?

great. ill update here (if i can find the thread) or open a github issue if i run into the same problem on the next update so ya'll can track it better