vip no route to host

[Dialgatrainer02]

Support requests should be sent via https://groups.io/g/keepalived-users

Describe why you are unable to send the support request to the above email list
Understanding why you cannot use the email list should help us improve it.
cant figure out how to make a new group for my issue and general unfamiliarity with mailing lists

Describe what you need help/support for
probably a config error

Details of what you would like to do with keepalived
Describe in details what you would like to achieve with keepalived
i have a 3 vault nodes in a cluster and i want virtual ip to point to a node so if goes down a new leader can be created without having to change vault adresses

Keepalived version
Output of keepalived -v (a later version of keepalived might be needed).

keepalived -v
Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+

Copyright(C) 2001-2023 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.14.0
Running on Linux 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z)
Distro: AlmaLinux 9.0 (Emerald Puma)

configure options: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-snmp --enable-snmp-rfc --enable-nftables --disable-iptables --enable-sha1 --enable-json --with-init=systemd build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig CC=gcc CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection LDFLAGS=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 

Config options:  NFTABLES LVS VRRP VRRP_AUTH VRRP_VMAC JSON OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd SYSTEMD_NOTIFY

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

Distro (please complete the following information):

• Name alma linux
• Version 9
• Architecture amd64

Details of any containerisation or hosted service (e.g. AWS)
running inside lxc on proxmox

Configuration file:
Full copy of your configuration file, obfuscated if necessary to protect passwords and IP addresses
leader node

vrrp_track_process track_vault {
    process vault
    delay 1
}

vrrp_instance VI_1 {
    state MASTER
    nopreempt
    interface eth0
    virtual_router_id 101
    priority  200
    advert_int 1
    authentication {
        auth_type AH
        auth_pass 12345
    }
    virtual_ipaddress {
        192.168.0.200
    }
    track_process {
        track_vault
    }
}

vrrp_track_process track_vault {
    process vault
    delay 1
}

vrrp_instance VI_1 {
    state BACKUP
    nopreempt
    interface eth0
    virtual_router_id 101
    priority  150
    advert_int 1
    authentication {
        auth_type AH
        auth_pass 12345
    }
    virtual_ipaddress {
        192.168.0.200
    }
    track_process {
        track_vault
    }
}

(they are generated with ansible )
Notify and track scripts
If any notify or track scripts are in use, please provide copies of them
tracking the vault process to see if its died

System Log entries
Full keepalived system log entries from when keepalived started, if applicable

Jun 22 21:08:18 vault2-test Keepalived[34237]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Jun 22 21:08:18 vault2-test Keepalived[34237]: Running on Linux 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) (built for Linux 5.14.0)
Jun 22 21:08:18 vault2-test Keepalived[34237]: Command line: '/usr/sbin/keepalived' '--dont-fork' '-D'
Jun 22 21:08:18 vault2-test Keepalived[34237]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 22 21:08:18 vault2-test Keepalived[34237]: Configuration file /etc/keepalived/keepalived.conf
Jun 22 21:08:18 vault2-test Keepalived[34237]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jun 22 21:08:18 vault2-test Keepalived[34237]: Starting VRRP child process, pid=34238
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Registering Kernel netlink reflector
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Registering Kernel netlink command channel
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: (VI_1) Initial state master is incompatible with AH authentication - clearing
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Assigned address 192.168.0.101 for interface eth0
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Assigned address fe80::be24:11ff:fe7b:e609 for interface eth0
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Failed to set/clear process event listen - errno 111 - Connection refused
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: (VI_1) entering FAULT state (tracked process track_vault quorum not achieved)
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: (VI_1) entering FAULT state
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: Registering gratuitous ARP shared channel
Jun 22 21:08:18 vault2-test Keepalived[34237]: Startup complete
Jun 22 21:08:18 vault2-test systemd[1]: Started LVS and VRRP High Availability Monitor.
░░ Subject: A start job for unit keepalived.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit keepalived.service has finished successfully.
░░ 
░░ The job identifier is 182709.
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: (VI_1) removing VIPs.
Jun 22 21:08:18 vault2-test Keepalived_vrrp[34238]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(51), fd(12,13) multicast, address(224.0.0.18)]

vault is definatly running so im unsure as to why its faulting
ps aux sinppet

vault       3290  0.0 29.5 211320568 155008 ?    Ssl  16:18   3:23 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

secondary node (also running a vault instance)

Jun 22 16:17:37 vault3-test systemd[1]: Starting LVS and VRRP High Availability Monitor...
░░ Subject: A start job for unit keepalived.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit keepalived.service has begun execution.
░░ 
░░ The job identifier is 1677.
Jun 22 16:17:37 vault3-test Keepalived[2718]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Jun 22 16:17:37 vault3-test Keepalived[2718]: Running on Linux 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) (built for Linux 5.14.0)
Jun 22 16:17:37 vault3-test Keepalived[2718]: Command line: '/usr/sbin/keepalived' '--dont-fork' '-D'
Jun 22 16:17:37 vault3-test Keepalived[2718]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 22 16:17:37 vault3-test Keepalived[2718]: Configuration file /etc/keepalived/keepalived.conf
Jun 22 16:17:37 vault3-test Keepalived[2718]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jun 22 16:17:37 vault3-test Keepalived[2718]: Starting VRRP child process, pid=2719
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Registering Kernel netlink reflector
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Registering Kernel netlink command channel
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Assigned address 192.168.0.102 for interface eth0
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Assigned address fe80::be24:11ff:fecb:f572 for interface eth0
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Failed to set/clear process event listen - errno 111 - Connection refused
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: (VI_1) entering FAULT state (tracked process track_vault quorum not achieved)
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: (VI_1) entering FAULT state
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Registering gratuitous ARP shared channel
Jun 22 16:17:37 vault3-test Keepalived[2718]: Startup complete
Jun 22 16:17:37 vault3-test systemd[1]: Started LVS and VRRP High Availability Monitor.
░░ Subject: A start job for unit keepalived.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit keepalived.service has finished successfully.
░░ 
░░ The job identifier is 1677.
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: (VI_1) removing VIPs.
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(51), fd(12,13) multicast, address(224.0.0.18)]

Additional context
i have a 3rd node which is basically identical to the second as they all use the same config with an jinja2 template
i can access each vault individually but cannot access vault through the vip
ping vip

PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data.
From 192.168.0.112 icmp_seq=1 Destination Host Unreachable
From 192.168.0.112 icmp_seq=2 Destination Host Unreachable
From 192.168.0.112 icmp_seq=3 Destination Host Unreachable

keepalived is definitely running

[Dialgatrainer02]

just having a look around and of the pid does this version still have the 32767 bug?
all of my instance pids are over 32767 so that might be a reason why
edit: yup my version is out of date and still has the bug

[pqarmitage]

The VRRP instance is in fault state and so the VIP 192.168.0.200 is not added to eth0. If you remove the vrrp_track_process (just for testing) you should find that you can ping 192.168.0.200. See the log entries:

Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: Failed to set/clear process event listen - errno 111 - Connection refused
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: (VI_1) entering FAULT state (tracked process track_vault quorum not achieved)
Jun 22 16:17:37 vault3-test Keepalived_vrrp[2719]: (VI_1) entering FAULT state

You appear to have built your own kernel (I have installed Alma Linux 9.4 in a VM and the kernel is 5.14.0). Is PROC_EVENTS enabled in your kernel? I suspect that the reason you are getting the Connection refused error is that your kernel is build without the proc_events connector.

[Dialgatrainer02]

I'm running lxc which uses the host kernel so I need to check if it's enabled or not also using a different track script worked curling vaults heath endpoint now my issue is vault not listening to the VIP but that's out of scope here