Here’s a weird issue I’m currently having. Any help would be greatly appreciated.
I am running Proxmox, but am having issues connecting to a service running in a VM from the host. For example, I am trying to post data from my Proxmox host and am sending it to the IP of the VM that is running on the host, but there seems to be connection issues somewhere. Everything works from from VM ↔ VM and from VM → Host. But not from Host → VM. Any ideas?
netcat returns “Connection refused” for the VM IP and port I’m trying to reach when trying to connect from the host. Has to be something related to the IP aliasing and iptables, but not sure how to resolve.
Seems I was wrong about being able to go from a VM to a service running on the host. Getting either “No route to host” or “Destination host unreachable”.
Here’s the route list. Let me know if you see anything strange. Note that I have never modified any iptable rules manually and have always just let Proxmox do it’s thing.
$ sudo ip route list
default via 107.174.26.x dev vmbr0 onlink
107.174.26.x-1/28 dev vmbr0 proto kernel scope link src 107.174.26.x+1
107.174.26.x-1/28 dev vmbr1 proto kernel scope link src 107.174.26.x+2
107.174.26.x-1/28 dev vmbr2 proto kernel scope link src 107.174.26.x+3
107.174.26.x-1/28 dev vmbr3 proto kernel scope link src 107.174.26.x+4
107.174.26.x-1/28 dev vmbr4 proto kernel scope link src 107.174.26.x+5
107.174.26.x-1/28 dev vmbr5 proto kernel scope link src 107.174.26.x+6
107.174.26.x-1/28 dev vmbr6 proto kernel scope link src 107.174.26.x+7
107.174.26.x-1/28 dev vmbr7 proto kernel scope link src 107.174.26.x+8
107.174.26.x-1/28 dev vmbr8 proto kernel scope link src 107.174.26.x+9
107.174.26.x-1/28 dev vmbr9 proto kernel scope link src 107.174.26.x+10
107.174.26.x-1/28 dev vmbr10 proto kernel scope link src 107.174.26.x+11
107.174.26.x-1/28 dev vmbr11 proto kernel scope link src 107.174.26.x+12
107.174.26.x-1/28 dev vmbr12 proto kernel scope link src 107.174.26.x+13
And here’s the route list from a container:
~$ sudo ip route list
default via 107.174.26.x dev eth0 proto dhcp src 107.174.26.x+2 metric 1024
107.174.26.x-1/28 dev eth0 proto kernel scope link src 107.174.26.x+2
107.174.26.x dev eth0 proto dhcp scope link src 107.174.26.x+2 metric 1024
Maybe I should be using a different subnet mask or something for the VM’s interfaces (like 255.255.255.255 or something to that effect).
That routing table looks strange. It think it’s saying that the 107.174.26.x-1/28 subnet can route through any of those adapters vmbr0 through vmbr12, which doesn’t seem right.
If you do want one separate bridge adapter per VM, I think you’d need to delete those routes and only route a /32 (ie. subnet mask 255.255.255.255, just that one IP) through the adapter.
I think normally you’d have a subnet assigned to just one bridge interface (eg. vmbr0) and then all the VMs would bridge to that single interface. That’s how I have it configured on one of my systems that uses LXD (albeit with private IPv4 addresses):
[email protected]:~$ sudo ip route list
default via xxx.xxx.xxx.xxx dev ens3
10.121.186.0/24 dev lxdbr0 proto kernel scope link src 10.121.186.1
10.123.1.0/24 dev tincvpn proto kernel scope link src 10.123.1.8
xxx.xxx.xxx.xxx/25 dev ens3 proto kernel scope link src xxx.xxx.xxx.xxx
Then I have three LXD containers bridged to lxdbr0.
Like I mentioned, I’ve never used Proxmox, so someone more familiar with it please feel free to correct me
I’ll attempt this on a fresh VM and unused bridge. Pretty sure I can change the mask in the web gui. Would rather not have to modify any of the iproutes directly so I don’t break my system and all my active VMs
But how Proxmox works is that each VM gets assigned the vmbr0 bridge, but you manually set the IP to one of the alias IPs. At least that’s what I’ve gathered from what I can see.
It also just dawned on me that I’m using isc-dhcp-server to give my VMs their network configs via DHCP. So there’s another possible location that the issue may reside. Will dig deeper and see
On my proxmox box with a /28 I have vmbr0 with the main IP, and vmbr1 with the first IP in the subnet. I don’t have a “bridge” for each VM. In my /etc/network/interfaces I’ve got one of up route add <IP FROM SUBNET>/32 dev vmbr0 per VM ip addresses (under the vmbr0 section)
It’s probably not correct as vmbr1 is the first usable IP in the subnet (and each VM uses it as the gateway) but it works.
I’d agree with the earlier comments on only routing the /32 to the VM.
@Mason did you get that solved? if not, let me know and I’ll have a look at your configs.
I think @Daniel is already on the right track here and you might have mixed up something between a routed and a bridged setup.
From my experience (depending on the provider) you can either use a /28 as a normal subnet with a single bridge, but are going to lose some IPs (brodcast, network etc.).
or do it with one bridge per IP to try and use all IPs for guests, but should then be using /32 subnet masks and pointopoint to not mess up the routing
Negative, I have not figure it out yet. I edited one of the Linux bridges to test. Changed the subnet mask to /32 and gave it a shot, but the VM had no connectivity. Doesn’t help that I’m trying to do this on a machine that’s actively being used for a plethora of things so I’m trying to avoid any restarts or network disruptions
Would I have had to restart the box or anything after changing the subnet mask? I was under the impression that any changes would take effect right away, but I could be wrong
yes, probably a reboot or at least restarting the network might be needed.
I do use a similar setup with one bridge per IP with an additional /29 with hetzner and can reach the guests via their external IP from the node properly. if you want, let’s have a look at your /etc/network/interfaces first