A way to exclude invalid items?

[kladnik]

Hi!

I am having trouble with solving an issue.
Is there a way to exclude the items that are not valid while not returning the error?
If we have an array of strings and one of them is null I'd like it to return the cleaned version of the array.

const array = ['test', 'test2', 'test3', null]
// Wanted behavior
const validatedArray = z.array(z.string()).parse(array) // ['test', 'test2', 'test3']

With very complex types this would be very practical. Currently I am looping over each item in array and checking, which seems very complicated especially once you're nested deeply.

Is there any better solution?

[scotttrinh]

I'm not aware of a way to adjust a Zod parser's behavior to get it to simply ignore items rather than throwing an error. You could write an array helper function and use preprocess maybe, but it's not going to be pretty since preprocess starts at unknown and requires you to do your own type narrowing.

If I was trying to implement a solution here, I would specify all of the types, or use z.array(z.any()) and use transform like this:

const array = ['test', 'test2', 'test3', null]
const validatedArray = z.array(z.any())
  .transform(
    (as) => as.filter((a) => z.string().safeParse(a).success)
  )
  .parse(array) // ['test', 'test2', 'test3']

Maybe this is a useful generalization?
CodeSandbox

const makeFilteredSchema = (s: z.ZodTypeAny) =>
  z.array(z.any()).transform(
    (as) => as.filter((a) => s.safeParse(a).success)
  );

const filteredStringArraySchema = makeFilteredSchema(z.string());
const filteredStringOrNumberSchema = makeFilteredSchema(z.union([z.string(), z.number()]);

// More likely used inline:

const fooSchema = z.object({
  bar: makeFilteredSchema(z.string()),
});

[kladnik]

Thanks! I did manage to do something similar with preprocess, not super pretty, but it works.
I do think a new option .exclude() on arrays and optional fields would be super practical though.

[scotttrinh]

Having truly heterogeneous arrays always feels like a smell to me, so I'm tempted to push back on the idea of including it as a part of the core library. If your array can contain string | null, I'd rather see the input schema reflect that and then use transform or a downstream filter to remove null or unexpected values. I think this keeps us a bit more aligned with TypeScript's concept of the Array type. But, you can certainly wrap Zod and include a ZodFilteredArray type as a custom type if you find a lot of use for it!

[flofehrenbacher]

Hello everybody, check out my preprocess solution with type narrowing -> https://gist.github.com/flofehrenbacher/71f93f7a423cfc11e1dd018325c241ec
Feedback is highly appreciated 🤓

[aurelticot]

I made a couple of tweaks to @scotttrinh's answer

const makeFilteredArraySchema = <T extends ZodSchema>(schema: T) =>
  z
    .array(z.unknown()) // `z.unknown()` more relevant than `z.any()` and avoid linting issue with any
    .transform((items) =>
      items?.filter(
        (item): item is z.infer<T> => schema.safeParse(item).success
        // `item is z.infer<T>` prevents the output type to be `unknown`. Ok to cast the type as it's been validated
      )
    );

const fooSchema = z.object({
  bar: makeFilteredArraySchema(z.string()),
});

const fooType = z.infer<typeof fooSchema>
// fooType = { bar: string[] } 

[SharmaTushar]

@aurelticot The return type doesn't seem be correctly inferred in one of my cases

import { z } from 'zod'
import { makeFilteredArraySchema } from './utils'

const menuBaseVariation = z.object({
    id: z.string(),
    title: z.string(),
})

const baseMenuItemSchema = z.object({
    model: z.string(),
    variations: menuBaseVariation.array(),
})

export type MenuItem = z.infer<typeof baseMenuItemSchema>['variations'][number] &
    (
        | { links?: MenuItem[] }
        | {
              type: string
              link: string
              opentype: string
          }
    )

export type MenuItemRaw = {
    model: string
    variations: ({
        id: string
        title: string
    } & (
        | { links?: MenuItemRaw[] }
        | {
              type: string
              link: string
              opentype?: string
          }
    ))[]
}

export const menuItemSchema: z.ZodType<MenuItem, z.ZodTypeDef, MenuItemRaw> = baseMenuItemSchema
    .extend({
        variations: makeFilteredArraySchema(
            z.union([
                menuBaseVariation.extend({
                    links: z.lazy(() => makeFilteredArraySchema(menuItemSchema).optional()),
                }),
                menuBaseVariation.extend({
                    type: z.string(),
                    link: z.string(),
                    opentype: z.string().optional(),
                }),
            ]),
        ),
    })
    .transform((item) => item.variations[0]!)

so I had to force TS to ignore it

export const makeFilteredArraySchema = <T extends z.ZodSchema>(schema: T): ReturnType<typeof z.array<T>> => {
    // @ts-expect-error
    return z
        .array(z.unknown()) // `z.unknown()` more relevant than `z.any()` and avoid linting issue with any
        .transform((items) => items?.filter((item): item is z.infer<T> => schema.safeParse(item).success))
}

Now for some reason, it's not correctly transforming the lazy items and instead of getting item.variations[0], I'm simply getting the item itself. Maybe the order of transformations has changed. I wonder how to preserve it.

Edit:
Using just filter transforms, but doesn't use the transformed result. A map step before filter has fixed it, maybe can be optimized to avoid looping over twice.

export const makeFilteredArraySchema = <T extends z.ZodSchema>(schema: T) => {
    return z
        .array(z.unknown()) // `z.unknown()` more relevant than `z.any()` and avoid linting issue with any
        .transform(
            (items) =>
                items
                    ?.map((item): z.infer<T> | undefined => {
                        const result = schema.safeParse(item)

                        if (result.success) return result.data
                        return undefined
                    })
                    .filter((item): item is z.infer<T> => item !== undefined),
        )
}

Edit:

I spoke too soon.

safeParse result has success true and data is undefined. Is this supposed to be happening?

[varunpvp]

This might help.

const zodArray = <S>(itemSchema: z.ZodType<S>) => {
  const catchValue = {} as never;

  return z
    .array(itemSchema.catch(catchValue))
    .transform((a) => a.filter((o) => o !== catchValue))
    .catch([]);
};

Example

const personSchema = z.object({
  name: z.string(),
  phones: zodArray(z.object({ label: z.string(), number: z.string() }))
});

const person = {
  name: "John",
  phones: [
    { number: "1234567890" }, // skipped
    { label: "Home" }, // skipped
    {}, // skipped
    { label: "Mobile", number: "1234567891" }
  ]
};

Output

{
  "success": true,
  "data": {
    "name": "John",
    "phones": [
      {
        "label": "Mobile",
        "number": "1234567891"
      }
    ]
  }
}

[lifeiscontent]

this seems to be the most complete implementation I could muster up:

const makeFilteredSchema = <S extends z.ZodTypeAny>(s: S) => {
  return z.array(z.unknown()).transform((as) => {
    const result: S[] = [];
    for (const a of as) {
      const parsed = s.safeParse(a);
      if (parsed.success) {
        result.push(parsed.data);
      }
    }
    return result;
  });
};

the reason why filter is not a great choice here, is because the schema itself could have additional transforms or maybe the data would have additional keys and in the case of filter they would not be removed, but because we're individually parsing each item, those additional transforms would be preserved.

[sorenhoyer]

@lifeiscontent I think it would still be relevant to many to be able to log the erroneous items that got removed, as zod issues ideally.

That seems to be the only thing missing.

[lifeiscontent]

@sorenhoyer can you give details in how you would use what you're expressing? I don't understand the use case.

[iiAku]

I believe what @sorenhoyer meant was a way to log out (at a certain level) what element are being filtered out from initial (pre-validation) list of items.

This solution works to me, I'm just struggling about the infered type when makeFilteredSchema is existing as part of a schema eg:

const mySchema = z.object({
  a: z.string(),
  b: makeFilteredSchema(anotherSchema)
})

Infered values of mySchema will return unknown due to z.array(unknown). Don't know how to make it infered properly.

[lifeiscontent]

@iiAku can you provide a minimal complete example? I can try to help but anotherSchema is hard to infer 😅

here's a basic idea of how to use makeFilteredSchema:

https://www.typescriptlang.org/play/?#code/JYWwDg9gTgLgBAbzgLzgXzgMyhEcBEyEAJvgNwBQFAxhAHYDO8IAhgNYCmAYsADYwcoHYgGVqACw6s4AXjgAeEXA4APAXWIMUAOgBaJACoBPMBwCCdIwD4AFAwBccEQEpZVxBThwhMAK5Q6HRYoKBYjG2RtXzo2OggAdzobZ2dtGFDGTGgQGxsWBlcZdwRPLzhaRnghBl9+RxEAbQBdWThmyjKsaDgbCqY4FjgITAGCj06vPvgwYIZhVoZtBhZMDgAFWY485w6J4BGbGag54iXfamoOBjGSibLq2phtMF8GcUPN0+IWGBYd0s6aAB6GBPn8gQe-F2aH+aEoNHo-QYEikgzkkQgACMAFYcagwGy3ZQhaCOSJMKDAOgAc2SABpSpCYA44KxODx+IJhGJJKwItpgqFwhicXiCUhVCxwLwOGSlukqbTXDCUgyYfCpl0IAsUaxnptCeqKEA

[iiAku]

I come up with a slightly different version (preprocess based) of yours and (at least to me) it's seems to work and infer better.

The problem I had was the inference were fine from the type when you create it from the schema even with your version.

However if you mimic what you would had with raw TS type and match it against that TS type is does not seems to be equal.

Here an example showing off what I am saying link

Here the same example using the version down below link

const makeFilteredSchema = <S extends z.ZodTypeAny>(s: S) => {
  return z.preprocess((as) => {
    const result: S[] = [];
    if (!Array.isArray(as)) {
      return result;
    }
    for (const a of as) {
      const parsed = s.safeParse(a);
      if (parsed.success) {
        result.push(parsed.data);
      }
    }
    return result;
  }, z.array(s));
};