Testing WebAuthn with Playwright.

[kentcdodds]

Hi there @MasterKale! Thank you so much for all of the awesome things that you're doing for the web authentication spec and libraries surrounding it. I really totally love this stuff. I'm just having a problem with automated testing. To make this report as useful as possible, I have recorded the problem in a live stream to demonstrate exactly the issue! It's at the start of this live stream here: https://www.youtube.com/watch?v=TV-HM4xVEdc

What I am demoing is just in the first 6 minutes.

The code for the passing test is here:

import { expect, test as setup } from '@playwright/test'

setup('partial passkey signup', async ({ page }) => {
	const client = await page.context().newCDPSession(page)
	await client.send('WebAuthn.enable')
	const { authenticatorId } = await client.send(
		'WebAuthn.addVirtualAuthenticator',
		{
			options: {
				protocol: 'ctap2',
				transport: 'usb',
				hasResidentKey: true,
				hasUserVerification: true,
				isUserVerified: true,
				automaticPresenceSimulation: true,
			},
		},
	)

	await page.goto('https://webauthn.io/')
	await page.getByPlaceholder('example_username').click()
	await page.getByPlaceholder('example_username').fill('test_user')
	await page.getByRole('button', { name: 'Register' }).click()

	await expect(page.getByText('Success!')).toBeVisible()

	const passkey = await client.send('WebAuthn.getCredentials', {
		authenticatorId,
	})

	expect(passkey.credentials).toHaveLength(1)

	let asserted = new Promise<void>((resolve) => {
		client.once('WebAuthn.credentialAsserted', () => resolve())
	})

	await page.getByRole('button', { name: 'Authenticate' }).click()

	await asserted

	await expect(page.getByText("You're logged in!")).toBeVisible()
})

The code for the failing test is here:

import { faker } from '@faker-js/faker'
import { expect, test } from '#tests/playwright-utils.ts'

async function setupWebAuthn(page: any) {
	const client = await page.context().newCDPSession(page)
	// https://chromedevtools.github.io/devtools-protocol/tot/WebAuthn/
	await client.send('WebAuthn.enable', { options: { enableUI: true } })
	const result = await client.send('WebAuthn.addVirtualAuthenticator', {
		options: {
			protocol: 'ctap2',
			transport: 'usb',
			hasResidentKey: true,
			hasUserVerification: true,
			isUserVerified: true,
			automaticPresenceSimulation: true,
		},
	})
	return { client, authenticatorId: result.authenticatorId }
}

test('Users can register and use passkeys', async ({ page, login }) => {
	const user = await login()

	const { client, authenticatorId } = await setupWebAuthn(page)

	const initialCredentials = await client.send('WebAuthn.getCredentials', {
		authenticatorId,
	})
	expect(
		initialCredentials.credentials,
		'No credentials should exist initially',
	).toHaveLength(0)

	await page.goto('/settings/profile/passkeys')

	const passkeyRegisteredPromise = new Promise<void>((resolve) => {
		client.once('WebAuthn.credentialAdded', () => resolve())
	})
	await page.getByRole('button', { name: /register new passkey/i }).click()
	await passkeyRegisteredPromise

	// Verify the passkey appears in the UI
	await expect(page.getByRole('list', { name: /passkeys/i })).toBeVisible()
	await expect(page.getByText(/registered .* ago/i)).toBeVisible()

	const afterRegistrationCredentials = await client.send(
		'WebAuthn.getCredentials',
		{ authenticatorId },
	)
	expect(
		afterRegistrationCredentials.credentials,
		'One credential should exist after registration',
	).toHaveLength(1)

	// Logout
	await page.getByRole('link', { name: user.name ?? user.username }).click()
	await page.getByRole('menuitem', { name: /logout/i }).click()
	await expect(page).toHaveURL(`/`)

	// Try logging in with passkey
	await page.goto('/login')
	const signCount1 = afterRegistrationCredentials.credentials[0].signCount

	const passkeyAssertedPromise = new Promise<void>((resolve) => {
		client.once('WebAuthn.credentialAsserted', () => resolve())
	})

	await page.getByRole('button', { name: /login with a passkey/i }).click()

	// Check for error message before waiting for completion
	const errorLocator = page.getByText(/failed to authenticate/i)
	const errorPromise = errorLocator.waitFor({ timeout: 1000 }).then(() => {
		throw new Error('Passkey authentication failed')
	})

	await Promise.race([passkeyAssertedPromise, errorPromise])

	// Verify successful login
	await expect(
		page.getByRole('link', { name: user.name ?? user.username }),
	).toBeVisible()

	// Verify the sign count increased
	const afterLoginCredentials = await client.send('WebAuthn.getCredentials', {
		authenticatorId,
	})
	expect(afterLoginCredentials.credentials).toHaveLength(1)
	expect(afterLoginCredentials.credentials[0].signCount).toBeGreaterThan(
		signCount1,
	)

	// Go to passkeys page and delete the passkey
	await page.goto('/settings/profile/passkeys')
	await page.getByRole('button', { name: /delete/i }).click()

	// Verify the passkey is no longer listed on the page
	await expect(page.getByText(/no passkeys registered/i)).toBeVisible()

	// But verify it still exists in the authenticator
	const afterDeletionCredentials = await client.send(
		'WebAuthn.getCredentials',
		{ authenticatorId },
	)
	expect(afterDeletionCredentials.credentials).toHaveLength(1)

	// Logout again to test deleted passkey
	await page.getByRole('link', { name: user.name ?? user.username }).click()
	await page.getByRole('menuitem', { name: /logout/i }).click()
	await expect(page).toHaveURL(`/`)

	// Try logging in with the deleted passkey
	await page.goto('/login')
	const deletedPasskeyAssertedPromise = new Promise<void>((resolve) => {
		client.once('WebAuthn.credentialAsserted', () => resolve())
	})

	await page.getByRole('button', { name: /login with a passkey/i }).click()

	await deletedPasskeyAssertedPromise

	// Verify error message appears
	await expect(page.getByText(/passkey not found/i)).toBeVisible()

	// Verify we're still on the login page
	await expect(page).toHaveURL(`/login`)
})

The code for the app we're running can be found here: https://github.com/epicweb-dev/epic-stack/tree/dev

The relevant bits can be found here: epicweb-dev/epic-stack@main...dev

I'm in the process of adding webauthn support to that repo, so this is a diff between what's currently running and what I added to support webauthn.

Thank you so much for your help! I'm very stuck!

[MasterKale]

Hello @kentcdodds, thanks for reaching out. I read through your issue and watched the video you linked; thanks for sending over the playwright test code too. Nothing immediately jumps out to me so I'll take a look in the next day or two and see if I can't propose a solution.

One thing I can recommend trying off-hand is, can you try adding a hard page refresh to your failing test after logging out? If your site is a SPA then in the past there's been some issues with certain browsers that didn't let you call WebAuthn twice without a hard page refresh after each call, which really dampened enthusiasm for SPA developers. That time should be long behind us, though, so I offer this only as something to try until I get back home to play around with your tests myself.

Can you confirm that your testing is running against macOS Chromium? And also what's that test runner you're using? I'm familiar with playwright but usually run them in the command line. I like the idea of a GUI-based test setup...

[MasterKale]

Bad news, pulling down the latest changes to dev branch didn't fix things. But good news, when I commented out SENTRY_DSN in .env then the site started loading just fine. Weird...anyway now I can investigate further.

[MasterKale]

So bizarre, after an hour or so of playing around with tests the fact that the overall high-level sequence of calls works fine on webauthn.io but fails against a locally-hosted site is unexplainable. A hard page reload and waiting 1000ms didn't fix anything so I'm guessing it's not some kind of weird WebAuthn virtual authenticator state issue. I kinda wonder if it's not some kind of bug in Chromium and Chrome DevTools Protocol...

I dug into how you're calling startAuthentication(). Compared to how I call it on webauthn.io it's sparse with just the single required challenge, but even when I expand your site's AuthenticationOptionsSchema so more options get passed in it has no positive impact on this issue.

To rule out @simplewebauthn/browser somehow obfuscating the root cause of your issue, I confirmed that this particular error is being passed directly through my library without any attempt to infer a cause:

Here's the raw NotAllowedError message that's returned from the actual call to navigator.credentials.get():

NotAllowedError: The operation either timed out or was not allowed. See: https://www.w3.org/TR/webauthn-2/#sctn-privacy-considerations-client.

I logged e.cause and e.code in the catch (e) around the call to startAuthentication() and confirmed it's the original error:

const errorMessage = getErrorMessage(e)
console.log('🤔🤔🤔', e.cause, e.code)
setError(`Failed to authenticate with passkey: ${errorMessage}`, e)

🤔🤔🤔 NotAllowedError: The operation either timed out or was not allowed. See: https://www.w3.org/TR/webauthn-2/#sctn-privacy-considerations-client. ERROR_PASSTHROUGH_SEE_CAUSE_PROPERTY

That all comes from here: https://github.com/MasterKale/SimpleWebAuthn/blob/master/packages/browser/src/helpers/identifyAuthenticationError.ts#L29-L38

Searching the Playwright issues for "webauthn" or "passkeys" pulled up the big one, microsoft/playwright#7276, none of the issues in its orbit offered any help.

I think this is about as far as I can take things on my own. I've backchanneled with one of my Chromium friends to see if maybe they have any idea, but I'm not holding my breath...I'll let you know if they offer any promising hints.

[MasterKale]

One thing that comes to mind: most of the examples I see of end-to-end use of Playwright's CDPSession to automate WebAuthn testing is against a live site. What if you ran your failing test against the dev site as hosted (temporarily) on a public URL? At this point that's the only thing left to check that might offer a clue as to whether or not this is a bug with CDPSession and localhost sites 🫠

[kentcdodds]

Thank you for trying. I thought something similar so dev is deployed to https://epic-stack-template-staging.fly.dev/

I had the same issue there too 😔

[MasterKale]

Okay, some progress: tweaking the Playwright config to launch against system chrome with headless: false...

projects: [
  {
    name: 'Google Chrome',
    use: {
      ...devices['Desktop Chrome'],
      channel: 'chrome',
      headless: false,
    },
  },
],

...I was able to open up chrome://device-log and see that the call to navigator.credentials.get() is failing because the virtual authenticator is reporting no credentials for the site:

FIDOError[06:49:23] Failing assertion request due to status 2 from VirtualFidoDevice-1CDCBB22E31A638C58F3A1B29724E4AB2023C5F35436A6DA06F0B40FA52863A4
FIDOError[06:49:23] CTAP error response code 46 received from VirtualFidoDevice-1CDCBB22E31A638C58F3A1B29724E4AB2023C5F35436A6DA06F0B40FA52863A4
FIDODebug[06:49:23] -> (CTAP2 error code 0x2e (kCtap2ErrNoCredentials))
FIDODebug[06:49:23] <- 0x2 (kAuthenticatorGetAssertion) {1: "localhost", 2: h'2AF6A6DD8BA574DE19D1C8A5AFDC8E18BB5BEF46A7B495127031F842F63E7F7E', 5: {"uv": true}}
FIDODebug[06:49:23] The device supports the CTAP2 protocol.
FIDODebug[06:49:23] -> {1: ["FIDO_2_0"], 3: h'01020304050607080102030405060708', 4: {"rk": true, "up": true, "uv": true, "plat": false}, 10: [{"alg": -8, "type": "public-key"}, {"alg": -7, "type": "public-key"}, {"alg": -257, "type": "public-key"}]}
FIDODebug[06:49:23] Sending CTAP2 AuthenticatorGetInfo request to authenticator.
FIDOEvent[06:49:23] Starting GetAssertion flow: { "allowCredentials": [ ], "challenge": "m1iHtEWviDiwVeHe5991seZ37Qd1mvG0SlqO_niRsgY", "rpId": "localhost", "userVerification": "preferred" }
FIDOError[06:49:23] Cannot test Bluetooth power status because process is not self-responsible. Launch from Finder to fix.
FIDOEvent[06:49:23] Enclave authenticator disabled because no suitable account
FIDODebug[06:49:22] -> {1: "packed", 2: h'49960DE5880E8C687434170F6476605B8FE4AEB9A28632C7995CF3BA831D97634500000001010203040506070801020304050607080020DA1071D2FE53CEBB1A92021B2655DF92952F9CD0249FC843B93B9D301C10A9BCA4010103272006215820BE6A59A4281E0AB413A866B7CE0988354E80FE1669481F2F19EB0BD6C28F572B', 3: {"alg": -7, "sig": h'3045022100B9B55271DB2EC7E5C8AD5AE83C8EE1BCCE6ECB25E77546609D477DF53B0D151A02202EE9BED62FA85F82B0A6CFED2B6DC993E5AD49C2EBFED9E88D93D9A0E400D956', "x5c": [h'308201D43082017AA003020102020101300A06082A8648CE3D0403023060310B30090603550406130255533111300F060355040A0C084368726F6D69756D31223020060355040B0C1941757468656E74696361746F72204174746573746174696F6E311A301806035504030C114261746368204365727469666963617465301E170D3137303731343032343030305A170D3435303231343134343932325A3060310B30090603550406130255533111300F060355040A0C084368726F6D69756D31223020060355040B0C1941757468656E74696361746F72204174746573746174696F6E311A301806035504030C1142617463682043657274696669636174653059301306072A8648CE3D020106082A8648CE3D030107034200048D617E65C9508E64BCC5673AC82A6799DA3C1446682C258C463FFFDF58DFD2FA3E6C378B53D795C4A4DFFB4199EDD7862F23ABAF0203B4B8911BA0569994E101A3253023300C0603551D130101FF040230003013060B2B0601040182E51C020101040403020520300A06082A8648CE3D04030203480030450221008F708E332E22D03861B3B0617AEAFA36DF8DA7386A966B6EFA6FF06B9ECC706102200DA99A9972A3A9D7C94F8900E04CF77B110A7B40083E89D3D4ABBF4C1C8049C3']}}
FIDODebug[06:49:22] <- 0x1 (kAuthenticatorMakeCredential) {1: h'87A1E909494C71A3848CEC4B43108810DD377A172CA63C64E058FB7BC68A8BA1', 2: {"id": "localhost", "name": "localhost"}, 3: {"id": h'636D37633134616D79303030306934656C6C75377376326A31', "name": "2v_fabiola_stokes0", "displayName": "Fabiola Stokes"}, 4: [{"alg": -8, "type": "public-key"}, {"alg": -7, "type": "public-key"}, {"alg": -257, "type": "public-key"}], 7: {"uv": true}}
FIDODebug[06:49:22] The device supports the CTAP2 protocol.
FIDODebug[06:49:22] -> {1: ["FIDO_2_0"], 3: h'01020304050607080102030405060708', 4: {"rk": true, "up": true, "uv": true, "plat": false}, 10: [{"alg": -8, "type": "public-key"}, {"alg": -7, "type": "public-key"}, {"alg": -257, "type": "public-key"}]}
FIDODebug[06:49:22] Sending CTAP2 AuthenticatorGetInfo request to authenticator.
FIDOEvent[06:49:22] Starting MakeCredential flow: { "attestation": "none", "challenge": "RPJTKwEXYw8HeozRyFVo9HODIXTcvQqcwi1upSzigu0", "excludeCredentials": [ ], "pubKeyCredParams": [ { "alg": -8, "type": "public-key" }, { "alg": -7, "type": "public-key" }, { "alg": -257, "type": "public-key" } ], "rp": { "id": "localhost", "name": "localhost" }, "user": { "displayName": "Fabiola Stokes", "id": "Y203YzE0YW15MDAwMGk0ZWxsdTdzdjJqMQ", "name": "2v_fabiola_stokes0" } }
FIDOError[06:49:22] Cannot test Bluetooth power status because process is not self-responsible. Launch from Finder to fix.
FIDOEvent[06:49:22] Enclave authenticator disabled because no suitable account

There's a line in there I see twice, once at the very bottom (the oldest log), then in the middle just before it handles the call to .get():

Enclave authenticator disabled because no suitable account

I wonder if this is a clue as to why it seems the virtual authenticator immediately forgets about the credential it just created

[MasterKale]

@kentcdodds I figured it out! It's this step in which you parse the registration options:

https://github.com/epicweb-dev/epic-stack/blob/edbb7945e68e9766895d49ad03c911d9ee1472b8/app/routes/settings%2B/profile.passkeys.tsx#L89

It's stripping out the authenticatorSelection options that include the residentKey property. Without this the virtual authenticator is creating non-discoverable credentials that cannot be used for passkey login flows (i.e. a call to startAuthentication()/navigator.credentials.get() with allowCredentials: [].)

Update RegistrationResultSchema (fwiw I'd recommend calling this RegistrationOptionsSchema to better communicate it's not the result of calling startRegistration() as the current name implied to me) to define the properties on authenticatorSelection (spec link) and you should get unblocked.

The only other thing I had to do to get that test to full pass is update the final getByText() to check for /failed to authenticate with passkey/i instead.

Give it a shot and let me know if it works!

[kentcdodds]

That totally worked. I'm so thrilled! Thank you so much for the help!