+-------------------------------------------------+------------+--------------+----------------+------------------------+ | Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | +-------------------------------------------------+------------+--------------+----------------+------------------------+ | file:/var/logs/homeassistant/home-assistant.log | 4 | - | 4 | - | | file:/var/logs/meshcentral/auth.log | 5 | - | 5 | - |
logline: Feb 13 15:39:30 meshcentral https[18]: Failed password for Jeroen from 172.18.0.200 port 58726

Its the same logline as in my tests so i dont know what i did wrong.

Never wasted time learning things

yes that is true! appreciated tho 🙂

nice find btw

@iiamloz still no cigar…
Logline: Feb 13 18:04:36 meshcentral https[18]: Failed password for jeredozajdj from 81.82.208.17 port 58560
Metrics:
Acquisition Metrics: +-------------------------------------------------+------------+--------------+----------------+------------------------+ | Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | +-------------------------------------------------+------------+--------------+----------------+------------------------+ | file:/var/logs/homeassistant/home-assistant.log | 26 | - | 26 | - | | file:/var/logs/meshcentral/auth.log | 20 | - | 20 | - |

What does your acquis.yaml look like?

or the full file?

Full in case you need it
`
filenames:

• /var/logs/default-host_access.log
• /var/logs/proxy-host-*_access.log
• /var/logs/proxy-host-*_error.log
  labels:
  type: nginx-proxy-manager

filenames:

• /var/logs/nextcloud/nextcloud.log
  labels:
  type: Nextcloud

filenames:

• /var/logs/homeassistant/home-assistant.log
  labels:
  type: home-assistant

filenames:

• /var/logs/jellyfin/log_*.log
  labels:
  type: jellyfin

filenames:

• /var/logs/meshcentral/auth.log
  labels:
  type: meshcentral
  `

Its in a docker container, the real logs are ofc mounted, like this:
- /home/jeroen/docker/ssd-data/meshcentral/data/auth.log:/var/logs/meshcentral/auth.log:ro

they are readable by the container bc i can do this:
docker container exec crowdsec_lapi tail -f /var/logs/meshcentral/auth.log Feb 13 17:55:53 meshcentral mps[18]: Server listening on 0.0.0.0 port 4433. Feb 13 18:01:25 meshcentral http[18]: Server listening on 0.0.0.0 port 80. Feb 13 18:01:27 meshcentral mps[18]: Server listening on 0.0.0.0 port 4433. Feb 13 18:01:31 meshcentral https[18]: Accepted password for Jeroen from 10.30.52.182 port 32910 Feb 13 18:02:18 meshcentral https[18]: User Jeroen logout from 10.30.52.182 port 39106 Feb 13 18:02:23 meshcentral https[18]: Failed password for Jeroen from 10.30.52.182 port 39156 Feb 13 18:04:36 meshcentral https[18]: Failed password for jeredozajdj from 81.82.208.17 port 58560 Feb 13 18:07:14 meshcentral https[18]: Failed password for Jeroen from 10.30.52.182 port 54204 Feb 13 18:07:20 meshcentral https[18]: Accepted password for Jeroen from 10.30.52.182 port 46954 Feb 13 18:07:27 meshcentral https[18]: Failed password for Jeroen from 10.30.52.182 port 47670

@iiamloz all looks fine, no?

what happens if you run the following

cscli explain  -l "Feb 13 18:02:23 meshcentral https[18]: Failed password for Jeroen from 10.30.52.182 port 39156" -t meshcentral

or the actual log file

cscli explain  -f /var/logs/meshcentral/auth.log -t meshcentral

example output to show that it is working

line: Feb 13 18:02:23 meshcentral https[18]: Failed password for Jeroen from 10.30.52.182 port 39156
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 a1ad/meshcentral-logs (+7 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |       └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
        └-------- parser failure 🔴

line: Feb 13 19:01:10 meshcentral https[18]: Failed password for jeredozajdj from 81.82.208.17 port 53408 ├ s00-raw | ├ 🔴 crowdsecurity/docker-logs | ├ 🟢 crowdsecurity/non-syslog (first_parser) | └ 🔴 crowdsecurity/syslog-logs ├ s01-parse | ├ 🔴 LePresidente/jellyfin-logs | ├ 🔴 a1ad/meshcentral-logs | ├ 🔴 crowdsecurity/home-assistant-logs | ├ 🔴 crowdsecurity/nextcloud-logs | ├ 🔴 crowdsecurity/nginx-proxy-manager-logs | └ 🔴 crowdsecurity/sshd-logs └-------- parser failure 🔴

so what is the difference … with the test data ..
docker container exec crowdsec_lapi cscli explain -l "Feb 13 19:29:59 meshcentral https[18]: Failed password for vzefvzefze from 81.82.208.17 port 48964" -t meshcentral line: Feb 13 19:29:59 meshcentral https[18]: Failed password for vzefvzefze from 81.82.208.17 port 48964 ├ s00-raw | ├ 🔴 crowdsecurity/docker-logs | ├ 🟢 crowdsecurity/non-syslog (first_parser) | └ 🔴 crowdsecurity/syslog-logs ├ s01-parse | ├ 🔴 LePresidente/jellyfin-logs | ├ 🔴 a1ad/meshcentral-logs | ├ 🔴 crowdsecurity/home-assistant-logs | ├ 🔴 crowdsecurity/nextcloud-logs | ├ 🔴 crowdsecurity/nginx-proxy-manager-logs | └ 🔴 crowdsecurity/sshd-logs └-------- parser failure 🔴

i really don't get it

When i test a logline from production in the dev env, all is good:
Line not working in production:
Feb 13 19:29:59 meshcentral https[18]: Failed password for vzefvzefze from 81.82.208.17 port 48964
hubtest in dev:
results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Parsed["message"] == "Feb 13 19:29:59 meshcentral https[18]: Failed password for vzefvzefze from 81.82.208.17 port 48964" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Parsed["program"] == "meshcentral" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Parsed["source_ip"] == "81.82.208.17" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Parsed["timestamp"] == "Feb 13 19:29:59" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Parsed["username"] == "vzefvzefze" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["log_type"] == "meshcentral_failed_auth" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["service"] == "meshcentral" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["source_ip"] == "81.82.208.17" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["user"] == "vzefvzefze" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["datasource_path"] == "meshcentral-logs.log" results["s01-parse"]["a1ad/meshcentral-logs"][3].Evt.Meta["datasource_type"] == "file"

i am so lost right now…

How did you install the collection?

docker container exec crowdsec_lapi cscli collections install a1ad/meshcentral

same as the jellyfin one, and meshcentral is based on jellyfin

i'll try debug this today

kinda a weird one

works in linux wsl (My dev environment), docker it does nothing

So it looks like some parsers just break the flow and i have no idea why

brian@Brian:~$ sudo cscli explain -l "Feb 13 18:02:23 meshcentral https[18]: Failed password for jeroen from 10.30.52.182 port 39156" -t meshcentral
line: Feb 13 18:02:23 meshcentral https[18]: Failed password for jeroen from 10.30.52.182 port 39156
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 LePresidente/authelia-logs
        |       ├ 🔴 LePresidente/emby-logs
        |       └ 🟢 a1ad/meshcentral-logs (+7 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |       └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
        └-------- parser failure 🔴

brian@Brian:~$ sudo cscli parser install LePresidente/gitea-logs
INFO[14-02-2023 08:31:31] Ignoring file /etc/crowdsec/hub/collections/a1ad/meshcentral.yml of type collections
INFO[14-02-2023 08:31:31] Enabled parsers : LePresidente/gitea-logs
INFO[14-02-2023 08:31:31] Enabled LePresidente/gitea-logs
INFO[14-02-2023 08:31:31] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.
brian@Brian:~$ sudo cscli explain -l "Feb 13 18:02:23 meshcentral https[18]: Failed password for jeroen from 10.30.52.182 port 39156" -t meshcentral
line: Feb 13 18:02:23 meshcentral https[18]: Failed password for jeroen from 10.30.52.182 port 39156
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 LePresidente/authelia-logs
        |       ├ 🔴 LePresidente/emby-logs
        |       ├ 🔴 LePresidente/gitea-logs
        |       ├ 🔴 a1ad/meshcentral-logs
        |       └ 🔴 crowdsecurity/sshd-logs
        └-------- parser failure 🔴

but i see no reason why that parser should break it

Pinging @iiamloz just so he is aware as well.

ok so this is due to pattern_syntax: in the yaml files.

So if the same variable is defined in multiple parsers the first one is only used

i'll do a PR for all my parsers and make them unique based on the parser name

So for example:
jellyfin-logs.yaml

pattern_syntax:
  CUSTOMDATE: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"

meshcentral-logs.yaml

pattern_syntax:
  CUSTOMDATE: "%{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"

depending on the order the parsers go through, the first one read will be used for all parsers and doesn't get replaced

https://github.com/crowdsecurity/hub/pull/656

Yes this is the case because we have to load all syntaxs into the same parser context. I would prefer them to be locally scoped to the parser but the memory just got out of hand

Yea that sounds reasonable so made all mine unique for now just to avoid this in the future

So a custom date would be: CUSTOMMESHCENTRALDATE

oh yeah, i see it now in your PR

that is one hell of a "feature"

More of a limitation of the parser engine

Do i need to do something with the index json file?

We could work around it but when we tested it under load crowdsec used 3 times more ram

• when creating a PR

No that gets auto updated by repo

ok

@AES Think you need to pull from the main branch first

https://github.com/crowdsecurity/hub/pull/657

Myea

@iiamloz do i need to start over?

No just got to you branch and there should be an option to sync, but you have to resolve the conflicts

via github ^^

too late i guess 🙂

Just make sure you sync your main branch before creating a sub branch on your fork

yea i forgot

sorry about that

line: Feb 13 19:29:59 meshcentral https[18]: Failed password for vzefvzefze from 81.82.208.17 port 48964 ├ s00-raw | ├ 🔴 crowdsecurity/docker-logs | ├ 🟢 crowdsecurity/non-syslog (first_parser) | └ 🔴 crowdsecurity/syslog-logs ├ s01-parse | ├ 🔴 LePresidente/jellyfin-logs | ├ 🟢 a1ad/meshcentral-logs (+7 ~2) | └ 🔴 crowdsecurity/home-assistant-logs ├ s02-enrich | ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1) | ├ 🟢 crowdsecurity/geoip-enrich (+13) | ├ 🔴 crowdsecurity/http-logs | ├ 🔴 crowdsecurity/nextcloud-whitelist | └ 🟢 crowdsecurity/whitelists (unchanged) ├-------- parser success 🟢 ├ Scenarios ├ 🟢 a1ad/meshcentral-bf └ 🟢 a1ad/meshcentral-bf_user-enum

we have green lights 🙂

thanks @Lepresidente 👍

No problem, glad it was a easy fix