Deployment failed: "code": "PrivateIPAddressIsAllocated"

[bramvdklinkenberg]

Is this a request for help?: Yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one): Issue

---

What version of acs-engine?: Version: v0.9.4

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes v1.7.7

What happened:
When trying to deploy a k8s cluster I get the error:

Deployment failed. Correlation ID: 750a63f1-84f4-423f-851c-a72b5edbec69. {
"error": {
"code": "PrivateIPAddressIsAllocated",
"message": "IP configuration /subscriptions/0746e4ac-1483-4bf9-a1da-1a27bf6a7525/resourceGroups/RG20TESTBRAM/providers/Microsoft.Network/networkInterfaces/k8s-master-41581036-nic-0/ipConfigurations/ipconfig1 is using the private IP address 10.40.12.198 which is already allocated to resource /subscriptions/0746e4ac-1483-4bf9-a1da-1a27bf6a7525/resourceGroups/RG20TESTBRAM/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-41581036-nic-0.",
"details": []
}
}

What you expected to happen:
My k8s cluster to deploy and no ip conflicts while spinning up resources

How to reproduce it (as minimally and precisely as possible):
acs-engine generate kubernetes.json

az group deployment create --output json --name testbram --resource-group RG20TESTBRAM --template-file _output/bramtest/azuredeploy.json --parameters _output/bramtest/azuredeploy.parameters.json

The json file I used to generate the arm template is
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"kubernetesConfig": {
"networkPolicy": "azure"
},
"orchestratorVersion": "1.7.7"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "bramtest",
"vnetSubnetId": "/subscriptions/<<SUB_ID>>/resourceGroups/RG20BASE100/providers/Microsoft.Network/virtualNetworks/VN20DST100/subnets/SN20DST100-TESTBRAM",
"vmSize": "Standard_D2_v2",
"storageProfile" : "ManagedDisks",
"firstConsecutiveStaticIP": "10.40.12.198"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 1,
"vmSize": "Standard_D2_v2",
"storageProfile" : "ManagedDisks",
"vnetSubnetId": "/subscriptions/<<SUB_ID>>/resourceGroups/RG20BASE100/providers/Microsoft.Network/virtualNetworks/VN20DST100/subnets/SN20DST100-TESTBRAM",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "kubadmin",
"ssh": {
"publicKeys": [
{
"keyData": "${KEY}"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "${ID}",
"secret": "${SECRET}"
}
}
}

Anything else we need to know:
Upfront I created the resource group and I am using an existing vnet & subnet with range
10.40.12.192 - 10.40.12.255 (64 addresses).

When I use 3 agents and 3 masters then it creates all 3 agents (0, 1 and 2) and only 2 masters (1 and 2) and eventually I will then get the error:
Deployment failed. Correlation ID: 2946df95-26cf-4e34-b0f9-e2cda8ffee16. {
"error": {
"code": "PrivateIPAddressIsAllocated",
"message": "IP configuration /subscriptions/<<SUB_ID>>/resourceGroups/RG20TESTBRAM/providers/Microsoft.Network/networkInterfaces/k8s-master-41581036-nic-0/ipConfigurations/ipconfig1 is using the private IP address 10.40.12.198 which is already allocated to resource /subscriptions/0<<SUB_ID>>/resourceGroups/RG20TESTBRAM/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-41581036-nic-1.",
"details": []
}
}

[tamilmani1989]

@bramvdklinkenberg We need subnets that are atleast /23 or more (if more agents are needed). The problem here is by default each node (including master) needs 30 addresses for pods. So the subnet space(64 addresses) you have created won't be enough for 6 nodes (3 agents and 3 masters). The ip address 10.40.12.198 is allocated to one of agent nodes and there are no free addresses to assign to master node.
I suggest to unblock, please use large enough subnet, and specify first consecutive IP address that is towards the end of subnet range.
We should take a work item to improve this behavior.

[tamilmani1989]

@bramvdklinkenberg Also specify "vnetCidr" in master profile. This should be the vnet address space you have created.
eg: "masterProfile": {
"vnetCidr":"10.0.0.0/8"
}

[bramvdklinkenberg]

@tamilmani1989 that are big subnets then.... gonna check later today if this fixes the issue :).

[cloudopsguy]

I have encountered the same problem and I don't understand how this is related to subnet size. There seems to be a different issue with the master NIC allocation. If noted from above 10.40.12.198 is the IP that is trying to be reused while it is also specified as "firstConsecutiveStaticIP": "10.40.12.198" from the deployment API.

[tristanstraub]

I had a similar problem and found that the commit making "CNI turned on by default" caused many many NIC's to be created, exhausting the subnet IP space. Going back to tag v0.10.0 resolved this issue for me.

[sharmasushant]

@tristanstraub I am assuming you mean many ipconfig's to be created (and not NICs). This is by design. The containers are now part of the Vnet, and hence need IP addresses from the Vnet.

[tristanstraub]

@sharmasushant I'd have to double check this as it's been a month since I had this issue, but I'm fairly sure from memory that many NIC's were created, with each one using up an IP address from the subnet.

[gmagniez]

Hi,
We faced a similar issue, but with using a subnet for each type of node, one for masters (3 masters) and one other for workers node.
In fact it seems that the deployment try to allocate the 2nd an 3rd masters nic after the first one has allocated its dynamic range, i.e. firstConsecutiveStaticIP: 192.168.0.10, masters will need 0.10, 0.11 and 0.12 but the dynamic pool of the first master has already reserved from 0.4 to 0.34.
So with 3 masters and ipAddressCount of 30 (default value), firstConsecutiveStaticIP has to be at least around 192.168.0.95.
I have solved the issue by setting firstConsecutiveStaticIP at the end of the subnet range (i.e. 192.168.0.200)
It seems to be related to merge request #1966

Doc here: https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/features.md#custom-vnet

[tamilmani1989]

@gmagniez -with this merge #1966 make sure that enough IPs left beyond firstConsecutiveStaticIP for all master pods.

If you check the link https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/features.md#custom-vnet

we would have specified,
"As a result, for the master nodes, enough IP addresses (equal to ipAddressCount) should be available beyond firstConsecutiveStaticIP"

[bramvdklinkenberg]

I created a /23 (16.0 and 17.0 )network and set the firstConsecutiveStaticIP to 17.1... worked fine like that. So yes make sure you pick a ip address further down the range but with enough ip addresses left for the masters nodes including the ip's reserved by them.