IP table lock issues

[yahalomimaor]

I have deployed node-local-dns-cache in my cluster, version 1.22.28
its being running for couple of days and all seemed to be fine.
yesterday i got an issue which one of the pod got these errors:
[the pod was running for a about 30 min]

Can't lock /run/xtables.lock: Resource temporarily unavailable

[ERROR] Error checking/adding iptables rule {filter OUTPUT [-p udp -s 101.1.0.1 --sport 53 -j ACCEPT -m comment --comment NodeLocal DNS Cache: allow DNS traffic]}, error - error checking rule: exit status 4: Ignoring deprecated --wait-interval option.

Another app is currently holding the xtables lock. Stopped waiting after 5s

Seems like it cannot update the ip tables somehow, i believe this is something related to Kube-proxy which also try to update the Ip tables.( again this is a guess im not sure)
I know there were these kind of issues before, but couldn't see any valid solution

I have tried to restart the node-cache pods, but it crashed because it as unable to update the ip table.

Couple of questions:

1. Does the node-local-dns-cache check the iptables oftenly ( i can see in the logs that it only happens when the pod is starting)
2. Is there a way to increase the retries if the table is locked so the pod wont get crashed (maybe some flag or something)
3. if node-local-dns-cache fails, why other pods were unable to resolve via the coredns is fallback?

Thanks.

[zhangguanzhang]

I looked at the source code and found that the -w option is used to obtain the lock when using iptables. The problem is that other processes have obtained the iptables lock for too long and have not released it. The following information is needed

cat /etc/os-release
uname -a
iptables -V

[yahalomimaor]

@zhangguanzhang thanks for the reply,
I will check it once it happen again.
But still, if the node-local-cache is already running for some period of time let's say 30 min,
why would it get this kind of error, is there are any other operations in background which need access to the iptables?
(the pod wasn't restarted or had any errors)

[zhangguanzhang]

kube-proxy and kubelet

[yahalomimaor]

And it does make sense the node-local-cache will need access to ip tables even 30 min after he started?
As far as i saw it happens only in the init phase, correct me if im wrong

[zhangguanzhang]

Please note that 256-260

dns/cmd/node-cache/app/cache_app.go

Lines 241 to 260 in fa0192f

	func (c *CacheApp) runPeriodic() {
	// if a pidfile is defined in flags, setup iptables as soon as it's created
	if c.params.Pidfile != "" {
	for {
	if isFileExists(c.params.Pidfile) {
	break
	}
	clog.Infof("waiting for coredns pidfile '%s'", c.params.Pidfile)
	time.Sleep(time.Second * 1)
	}
	// we found the pidfile, coreDNS is running, we can setup networking early
	c.setupNetworking()
	}
	c.exitChan = make(chan struct{}, 1)
	tick := time.NewTicker(c.params.Interval * time.Second)
	for {
	select {
	case <-tick.C:
	c.setupNetworking()

in line 260 goto L208

dns/cmd/node-cache/app/cache_app.go

Lines 205 to 208 in fa0192f

	func (c *CacheApp) setupNetworking() {
	if c.params.SetupIptables {
	for _, rule := range c.iptablesRules {
	exists, err := c.iptables.EnsureRule(utiliptables.Prepend, rule.table, rule.chain, rule.args...)

need Increase the time of c.params.Interval

[yahalomimaor]

Got it,
Thank you very much @zhangguanzhang !!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten