Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] DNS Resolvement wont work inside Windows container #100

Closed
dsfrederic opened this issue Mar 30, 2023 · 10 comments
Closed

[BUG] DNS Resolvement wont work inside Windows container #100

dsfrederic opened this issue Mar 30, 2023 · 10 comments
Assignees
@fcabrera23
Labels
bug Something isn't working

Comments

@dsfrederic
Copy link

Describe the bug
In pod no DNS resolvement vs DNS resolvement on node.

Expected behavior
DNS resolves to same IP in pod as with Invoke AKS Edge node command.

Screenshots
image

Environment (please complete the following information):

Additional context
Followed the Azure ARC quickstart.
@dsfrederic dsfrederic added the bug Something isn't working label Mar 30, 2023
@dsfrederic dsfrederic changed the title [BUG] [BUG] DNS Resolvement Mar 30, 2023
@fcabrera23 fcabrera23 self-assigned this Apr 3, 2023
@fcabrera23
Copy link

Hi @dsfrederic,

Thanks for reaching out. Please provide more context around your deployment.

  1. What version of Windows are you using? Client or Server?
  2. What version of AKS EE are you using?
  3. Scalable or SingleMachine cluster?
  4. K3s or K8s?
  5. Are you able to resolve any other address from inside the pod? What about pinging an IP address?

Thanks,
Francisco

@dsfrederic
Copy link
Author

Windows Server 22

1.1.80

SingleMachine

K3S

No not possible. The weird thing is that it's able to pull docker containers but once in the container there's no network connection.

@dsfrederic
Copy link
Author

dsfrederic commented Apr 5, 2023 •
edited
Loading

This issue on the quickstart repo might be clarifying.

microsoft/azure_arc#1759 (comment)

@fcabrera23 do you guys have a workaround?

@fcabrera23
Copy link

Hi @dsfrederic,

Could you please try running the following commands in the 3 nodes? We've encountered a firewall issue that will be fixed in our next release.

Invoke-AksEdgeNodeCommand -NodeType Linux "sudo iptables -A INPUT -p udp --dport 8472 -j ACCEPT
Invoke-AksEdgeNodeCommand -NodeType LInux "sudo iptables-save | sudo tee /etc/systemd/scripts/ip4save"

Thanks,
Francisco

@dsfrederic
Copy link
Author

I've executed the commands you've sent me. But this doesn't seem to resolve my issue.

Output below:

PS C:\Users\arcdemo\Desktop> Invoke-AksEdgeNodeCommand -NodeType Linux "sudo iptables -A INPUT -p udp --dport 8472 -j ACCEPT"
PS C:\Users\arcdemo\Desktop> Invoke-AksEdgeNodeCommand -NodeType LInux "sudo iptables-save | sudo tee /etc/systemd/scripts/ip4save"
# Generated by iptables-save v1.8.7 on Fri Apr  7 15:28:32 2023
*mangle
:PREROUTING ACCEPT [44307348:6839246413]
:INPUT ACCEPT [44300790:6837978542]
:FORWARD ACCEPT [202:13584]
:OUTPUT ACCEPT [44322316:6847835018]
:POSTROUTING ACCEPT [44322458:6847845482]
:KUBE-IPTABLES-HINT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT
# Completed on Fri Apr  7 15:28:32 2023
# Generated by iptables-save v1.8.7 on Fri Apr  7 15:28:32 2023
*nat
:PREROUTING ACCEPT [109:21002]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2137:128505]
:POSTROUTING ACCEPT [2137:128505]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-6KJXLLADRY4FRUOJ - [0:0]
:KUBE-SEP-CQRAHCCC7XPEWYB2 - [0:0]
:KUBE-SEP-VOJ5KSDSMKF2NGVM - [0:0]
:KUBE-SEP-YP32F2HEJI64ZYJN - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/24 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SEP-6KJXLLADRY4FRUOJ -s 10.42.0.2/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-6KJXLLADRY4FRUOJ -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.42.0.2:9153
-A KUBE-SEP-CQRAHCCC7XPEWYB2 -s 192.168.0.2/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-CQRAHCCC7XPEWYB2 -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 192.168.0.2:6443
-A KUBE-SEP-VOJ5KSDSMKF2NGVM -s 10.42.0.2/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-VOJ5KSDSMKF2NGVM -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SEP-YP32F2HEJI64ZYJN -s 10.42.0.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-YP32F2HEJI64ZYJN -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp -> 10.42.0.2:53" -j KUBE-SEP-YP32F2HEJI64ZYJN
-A KUBE-SVC-JD5MR3NA4I4DYORP ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics -> 10.42.0.2:9153" -j KUBE-SEP-6KJXLLADRY4FRUOJ
-A KUBE-SVC-NPX46M4PTMTKRN6Y ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https -> 192.168.0.2:6443" -j KUBE-SEP-CQRAHCCC7XPEWYB2
-A KUBE-SVC-TCOU7JCQXEZGVUNU ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 10.42.0.2:53" -j KUBE-SEP-VOJ5KSDSMKF2NGVM
COMMIT
# Completed on Fri Apr  7 15:28:32 2023
# Generated by iptables-save v1.8.7 on Fri Apr  7 15:28:32 2023
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2379 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2380 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
-A INPUT -p udp -m udp --dport 8472 -j ACCEPT
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -s 10.42.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -j ACCEPT
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Fri Apr  7 15:28:32 2023

@dsfrederic
Copy link
Author

You're also referring to execute this on 3 nodes but as stated in the initial bug report I'm running a singleMachine cluster

@fcabrera23
Copy link

Hi @dsfrederic,

Thanks for your update. We're still investigating this issue, and we believe that it is related to the flannel plugin. Could you please try with our K8S version + Calico?

Thanks,
Franicsco

@fcabrera23 fcabrera23 changed the title [BUG] DNS Resolvement [BUG] DNS Resolvement wont work inside Windows container Apr 11, 2023
@dsfrederic
Copy link
Author

@fcabrera23 in the k8s version it doesn't seem to be a problem

@fcabrera23
Copy link

Hi @dsfrederic,

Thanks for the confirmation. We are working to fix this issue as soon as possible, and I'll provide a workaround once we have it fixed.

Thanks,
Francisco

@fcabrera23
Copy link

Hi @dsfrederic

Closing this issue - It was fixed with 1.2.414.0 release.

Thanks,
Francisco

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fcabrera23 fcabrera23

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fcabrera23 @dsfrederic