Netgear A8000 kernel oops on 6.12.13 (6.12.13 had a bug, patch is going in)

[salahcoronya]

Since upgrading to 6.12.13, I got the following OOPS:

[   32.098574] BUG: kernel NULL pointer dereference, address: 0000000000000400
[   32.098620] #PF: supervisor read access in kernel mode
[   32.098634] #PF: error_code(0x0000) - not-present page
[   32.098647] PGD 0 P4D 0 
[   32.098665] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
[   32.098683] CPU: 0 UID: 0 PID: 470 Comm: mt76-usb-rx phy Not tainted 6.12.13-gentoo-dist #1
[   32.098703] Hardware name:  /AMD HUDSON-M1, BIOS 4.6.4 11/15/2011
[   32.098717] RIP: 0010:mt792x_rx_get_wcid+0x48/0x140 [mt792x_lib]
[   32.098752] Code: fe 40 04 77 56 48 81 c5 84 06 00 00 48 8b 44 eb 08 48 85 c0 74 26 84 d2 75 22 f6 80 bc 00 00 00 01 74 24 48 8b 80 78 03 00 00 <48> 8b 80 00 04 00 00 48 85 c0 74 11 48 05 e0 00 00 00 48 83 c4 08
[   32.098776] RSP: 0018:ffffa147c055fd98 EFLAGS: 00010202
[   32.098792] RAX: 0000000000000000 RBX: ffff8e9ecb652000 RCX: 0000000000000000
[   32.098806] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8e9ecb652000
[   32.098819] RBP: 0000000000000685 R08: ffff8e9ec6570000 R09: 0000000000000000
[   32.098832] R10: ffff8e9ecd2ca000 R11: ffff8e9f22a217c0 R12: 0000000038010119
[   32.098845] R13: 0000000080843801 R14: ffff8e9ec6570000 R15: ffff8e9ecb652000
[   32.098860] FS:  0000000000000000(0000) GS:ffff8e9f22a00000(0000) knlGS:0000000000000000
[   32.098876] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   32.098889] CR2: 0000000000000400 CR3: 000000000d2ea000 CR4: 00000000000006f0
[   32.098903] Call Trace:
[   32.098918]  <TASK>
[   32.098932]  ? __die_body.cold+0x19/0x27
[   32.098955]  ? page_fault_oops+0x15a/0x2f0
[   32.098975]  ? search_module_extables+0x19/0x60
[   32.098995]  ? search_bpf_extables+0x5f/0x80
[   32.099012]  ? exc_page_fault+0x7e/0x180
[   32.099030]  ? asm_exc_page_fault+0x26/0x30
[   32.099054]  ? mt792x_rx_get_wcid+0x48/0x140 [mt792x_lib]
[   32.099084]  mt7921_queue_rx_skb+0x1c6/0xaa0 [mt7921_common]
[   32.099114]  mt76u_alloc_queues+0x784/0x810 [mt76_usb]
[   32.099140]  ? __pfx___mt76_worker_fn+0x10/0x10 [mt76]
[   32.099172]  __mt76_worker_fn+0x4f/0x80 [mt76]
[   32.099203]  kthread+0xd2/0x100
[   32.099221]  ? __pfx_kthread+0x10/0x10
[   32.099237]  ret_from_fork+0x34/0x50
[   32.099254]  ? __pfx_kthread+0x10/0x10
[   32.099269]  ret_from_fork_asm+0x1a/0x30
[   32.099290]  </TASK>
[   32.099300] Modules linked in: 8021q garp mrp nf_conntrack_netbios_ns nf_conntrack_broadcast nft_masq nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security bridge stp llc nf_tables ip6table_filter ip6_tables iptable_filter ip_tables amdgpu bnep amdxcp gpu_sched drm_exec drm_buddy at24 kvm_amd mt7921u mt792x_usb mt7921_common rt2800pci rt2800mmio mt792x_lib rt2800lib rt2x00pci mt76_connac_lib rt2x00mmio mt76_usb rt2x00lib mt76 kvm r8153_ecm cdc_ether usbnet mac80211 pcspkr acpi_cpufreq r8169 eeprom_93cx6 k10temp realtek libarc4 usblp snd_hda_codec_realtek btusb snd_hda_codec_generic btrtl snd_hda_scodec_component i2c_piix4 btintel snd_hda_codec_hdmi snd_hda_intel i2c_smbus btbcm btmtk snd_intel_dspcfg radeon snd_intel_sdw_acpi i2c_algo_bit cfg80211 snd_hda_codec bluetooth
[   32.099538]  drm_suballoc_helper drm_ttm_helper r8152 mii snd_hda_core rfkill ttm snd_hwdep drm_display_helper cec snd_pcm video wmi snd_timer snd soundcore vfat fat joydev loop fuse nfnetlink sha512_ssse3 sha256_ssse3 ata_generic pata_acpi sha1_ssse3 serio_raw sp5100_tco pata_atiixp
[   32.099706] CR2: 0000000000000400
[   32.099719] ---[ end trace 0000000000000000 ]---
[   32.099730] RIP: 0010:mt792x_rx_get_wcid+0x48/0x140 [mt792x_lib]
[   32.099758] Code: fe 40 04 77 56 48 81 c5 84 06 00 00 48 8b 44 eb 08 48 85 c0 74 26 84 d2 75 22 f6 80 bc 00 00 00 01 74 24 48 8b 80 78 03 00 00 <48> 8b 80 00 04 00 00 48 85 c0 74 11 48 05 e0 00 00 00 48 83 c4 08
[   32.099781] RSP: 0018:ffffa147c055fd98 EFLAGS: 00010202
[   32.099795] RAX: 0000000000000000 RBX: ffff8e9ecb652000 RCX: 0000000000000000
[   32.099808] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8e9ecb652000
[   32.099821] RBP: 0000000000000685 R08: ffff8e9ec6570000 R09: 0000000000000000
[   32.099833] R10: ffff8e9ecd2ca000 R11: ffff8e9f22a217c0 R12: 0000000038010119
[   32.099846] R13: 0000000080843801 R14: ffff8e9ec6570000 R15: ffff8e9ecb652000
[   32.099858] FS:  0000000000000000(0000) GS:ffff8e9f22a00000(0000) knlGS:0000000000000000
[   32.099873] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   32.099885] CR2: 0000000000000400 CR3: 000000000d2ea000 CR4: 00000000000006f0
[   32.099898] note: mt76-usb-rx phy[470] exited with irqs disabled

I have 2 of these adapters, both in AP mode - one is a 5 GHz AP, the other a 6GHz AP. Hostapd is still responsive but cannot be killed even with SIGKILL. 2.4 and 5 GHz continue to work, but 6 GHz does not. The first shutdown/unplug of the network adapters did not resolve it, but the 2nd try worked.

[morrownr]

Hi @salahcoronya

I have a couple of adapters that use the same chip as your Netgear A8000... mt7921au. I am using kernel 6.12.09. I just checked and am not seeing what you are seeing. This is on a Debian 12 system with a backported kernel.

What you might consider doing for now is going back to the previous kernel that was working okay. Then give kernel 6.12 a little more time in the over to see if it gets cooked.

I see a null pointer dereference but something doesn't seem to make sense to me. I'll keep an eye open.

[paulmenzel]

@salahcoronya, what was the last working Linux version?

It’d be great if you could bisect to find the culprit.

[morrownr]

Another report. That did not take long.

#579

What was your previous kernel version?

Was it stable?

[salahcoronya]

Previous kernel version was stable, 6.12.12

[morrownr]

This is beginning to look like a patch that went into 6.12.13 caused this. As soon as I get a confirmation from the other person reporting the same thing, I plan to make a report to linux-wireless to see if we can get a Mediatek dev on this. I'm going to check kernel 6.14 to see if maybe it could be a patch in 6.14 that was just backported to 6.12.

Wireless is complicated these days. This is not WiFi 4 anymore.

[salahcoronya]

Just an update: I've been bisecting over the last 2 days. The oops usually doesn't happen immediately, but after about 4 hours of uptime, so this is going to take a while.

[morrownr]

Thanks for the update. A bisect showing exactly which patch caused this would be a great thing and I will use it in the report.

Keep me posted.

[salahcoronya]

The bisection is finally complete. It points to this commit:

3fe7acc6f4b42ccb1056c5847f18f8eb2fec0834 is the first bad commit
commit 3fe7acc6f4b42ccb1056c5847f18f8eb2fec0834
Author: Ming Yen Hsieh <mingyen.hsieh@mediatek.com>
Date:   Tue Dec 10 17:19:20 2024 -0800

    wifi: mt76: mt7925: Update mt792x_rx_get_wcid for per-link STA
    
    [ Upstream commit 90c10286b176421068b136da51ed83059a68e322 ]
    
    Update mt792x_rx_get_wcid to support per-link STA.
    
    Fixes: 86c051f2c418 ("wifi: mt76: mt7925: enabling MLO when the firmware supports it")
    Signed-off-by: Ming Yen Hsieh <mingyen.hsieh@mediatek.com>
    Signed-off-by: Sean Wang <sean.wang@mediatek.com>
    Link: https://patch.msgid.link/20241211011926.5002-11-sean.wang@kernel.org
    Signed-off-by: Felix Fietkau <nbd@nbd.name>
    Signed-off-by: Sasha Levin <sashal@kernel.org>

 drivers/net/wireless/mediatek/mt76/mt792x_mac.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

bisect.log

[morrownr]

Thanks @salahcoronya

Email sent to linux-wireless and others. I'll keep you informed.

[paulmenzel]

Thanks @salahcoronya

Email sent to linux-wireless and others. I'll keep you informed.

https://lore.kernel.org/linux-wireless/41eaf1f2-ea82-47a5-8586-60e3337f49bf@gmail.com/T/#u

I recommend to add regressions@lists.linux.dev to the recipients, and also make use of regzbot.

[morrownr]

@paulmenzel

I kinda forgot. Thanks.

[qqgnoe466263]

Hi,

Thanks for report this issue, I will look into this issue.
And does it happen as soon as AP mode is activated, or?

Thanks~

[salahcoronya]

Its not usually instant. it can take as long as 6 hours to appear.

[morrownr]

@qqgnoe466263

All 3 reports about this problem are using mt7921au based usb adapters with the mt7921u driver and all 3 are using AP mode.

[qqgnoe466263]

Hi,

Could you help to test this patch? I think it can fix this issue. Thanks~

diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/main.c b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
index 7b5ff1237e9d..5195925a2705 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7921/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
@@ -811,6 +811,7 @@ int mt7921_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif,
        msta->deflink.wcid.phy_idx = mvif->bss_conf.mt76.band_idx;
        msta->deflink.wcid.tx_info |= MT_WCID_TX_INFO_SET;
        msta->deflink.last_txs = jiffies;
+       msta->deflink.sta = msta;
 
        ret = mt76_connac_pm_wake(&dev->mphy, &dev->pm);
        if (ret)
 

[Simon566]

Hi,

is this patch from wireless next ? is there some information along with it ?

regards,
Simon

[qqgnoe466263]

Hi @Simon566,

I am the author of this commit (wifi: mt76: mt7925: Update mt792x_rx_get_wcid for per-link STA), so I am also responsible for fixing it. I believe this patch can resolve the issue. If everyone tests it and there are no problems, I will send it to wireless next.

[Simon566]

thx , i will test it

[salahcoronya]

I'm testing it too. It'll take 6 hours to verify if the oops occurs or not.

[salahcoronya]

6 hours have passed without an oops (on 6.12.14 + above patch), so it looks good.

[morrownr]

@qqgnoe466263

If everyone tests it and there are no problems, I will send it to wireless next.

May I suggest something? wireless-next might not be the appropriate place. A regression that is taking systems down should qualify to go directly to Linus (Lead maintainer for test kernel) and Greg (Lead maintainer for stable kernels). You might talk to Felix about this.

[qqgnoe466263]

@qqgnoe466263

If everyone tests it and there are no problems, I will send it to wireless next.

May I suggest something? wireless-next might not be the appropriate place. A regression that is taking systems down should qualify to go directly to Linus (Lead maintainer for test kernel) and Greg (Lead maintainer for stable kernels). You might talk to Felix about this.

Yes, i will do it, and add test tag with @Simon566 , @salahcoronya and you. Is that ok?

Thanks~

[qqgnoe466263]

Hi @Simon566 , @salahcoronya ,

Could you provide me with your email so I can fill out the "Tested-by"?

Thanks~

[salahcoronya]

It is salah.coronya@gmail.com

[morrownr]

I'm good for:

Reported-by: Nick Morrow USBWiFi2024@gmail.com

If necessary, I can test but I would need some time set up.

[qqgnoe466263]

I'm good for:

Reported-by: Nick Morrow USBWiFi2024@gmail.com

If necessary, I can test but I would need some time set up.

Oops... I have sent the patch...

I can send v2 for this.

[morrownr]

@qqgnoe466263

No, please don't do a v2 for this unless you think it might come in handy if you need to contact me. I really do appreciate your timely work and your company, Mediatek, is appreciated by many Linux USB WiFi adapter and module users. Mediatek is doing USB Linux drivers the right way. Thank You.

@morrownr

[qqgnoe466263]

@qqgnoe466263

No, please don't do a v2 for this unless you think it might come in handy if you need to contact me. I really do appreciate your timely work and your company, Mediatek, is appreciated by many Linux USB WiFi adapter and module users. Mediatek is doing USB Linux drivers the right way. Thank You.

@morrownr

ok. Thanks for you comment