Prepare for Fair user policy updates

[Simply007]

Brief bug description

From the 1st of February the /items endpoint is limited to 1000 (incl. linked items) as described in Kontent API changes.

After that date, queries resulting in more than 1K content items per language won't load more than that 1K items.

Expected behavior

There is an alternative endpoint introduced /items-feed.

Allow source plugin to configure, whether /items or /items-feed endpoint is used and describe the pros and cons in Readme.

Additional context

In the SDK, there is a method itemsFeedAll that allows the loading of all of the items.

• There is only one downside is that after this - the Embedded JS SDK resolution is not performed and resolution must be made manually according to the article, or by using the sample component.

[bwlng]

As you mentioned above, the plugin currently uses the /items endpoint.

It sounds like the /items-feed endpoint is the appropriate endpoint for a plugin like this that should consume all content within the project.

However, the lack of resolution will break our current site and the methods outlined in the linked articles and components won't work for two reasons:

1. We make heavy use of the resolvedUrl field to create links throughout our site, so if that data won't be available after switching to the /items-feed endpoint, we'd have to refactor a large portion of our site’s code
2. The data provided in the links field of a Rich Text element doesn’t contain enough data to reconstruct the URLs our content currently uses. For example, the URL Slug is provided, but if the item has a parent entry so that the resolved URL should be parent-url-slug/url-slug, there is no way to recreate the URL.

This is an example of a resolver that is common across content on our site:

import { ContentItem } from "@kentico/kontent-delivery"

module.exports = class Page extends ContentItem {
  constructor() {
    super({
      urlSlugResolver: (link, context) => {
        const entry = context.item
        const [parentElement] = entry ? entry.parent_element.value : []

        return {
          url: `${parentElement ? `/${parentElement.url_slug.value}` : ``}/${
            entry.url_slug.value
          }`,
        }
      },
    })
  }
}

In the case of rich text fields that link to nested content, the unresolved data would like something like this:

{
  "data": {
    "pages": {
      "elements": {
        "url_slug": {
          "resolvedUrl": "/support/evaluation-forms"
        },
        "body": {
          "value": "<p><strong>Evaluation Forms</strong></p>\n<ul>\n  <li>Spiral Form (<a data-item-id=\"dc284d7c-bcd7-4f42-8e93-774060aea8db\" href=\"\">imperial</a> or <a data-item-id=\"9d7226d1-6496-4102-889b-31b61ff99224\" href=\"\">metric</a>)</li></ul>",
          "links": [
            {
              "urlSlug": "spiral-imperial",
              "type": "page"
            },
            {
              "urlSlug": "spiral-metric",
              "type": "page"
            }
          ],
          "resolvedData": {
            "html": "<p><strong>Evaluation Forms</strong></p>\n<ul>\n  <li>Spiral Form (<a data-item-id=\"dc284d7c-bcd7-4f42-8e93-774060aea8db\" href=\"/support/evaluation-forms/spiral-imperial\">imperial</a> or <a data-item-id=\"9d7226d1-6496-4102-889b-31b61ff99224\" href=\"/support/evaluation-forms/spiral-metric\">metric</a>)</li></ul>"
          }
        },
      }
    }
  }
}

Where a resolved URL in the body looks like this /support/evaluation-forms/spiral-imperial the unresolved URL could only be recreated as /page/spiral-imperial.

To prepare for the Fair Usage Policy change:

Will the JS SDK be updated to handle resolving data for items-feed or will there be another method or option for sites with more than 1000 items to properly resolve data that was previously handled by the SDK?

[Simply007]

Hello @bwlng,
Thanks for the feedback. Let me first clarify the impact on this source plugin - since the content is loaded per language - the limitation will affect projects with 1000+ per language.

Implementing items-feed to gatsby source plugin is fairly simple, but the SDK itself currently does not offer item resolution in that endpoint.

We will take a look at your situation try to come up with the solution. I will keep you posted here.

[bwlng]

Thanks @Simply007 — this project is using the following versions of dependencies:

• @kentico/gatsby-source-kontent v4.0.0-beta2
• @kentico/kontent-delivery v8.0.0

[Simply007]

I have created a draft pull request that will fix the issue, once it is unblocked - that means the resolution in the SDK is implemented in items feed endpoint as well. Then the only thing required is the source plugin version update and the useItemsFeedEndpoint property setting in the gatsby-config.js file.

We are currently discussing the options we have to not affect the projects that are about to exceed the API limitations. I will keep you updated here!

Your 🥑

[Simply007]

Breaking news 📢

Thanks to the @Enngage and his work in implementing kontent-ai/delivery-sdk-js#275 we have decided to switch from /items endpoint to /items-feed endpoint with no breaking changes!

The only action required from you to prevent limitation hit is to update your package version to version 4.3 once it is released.

After the #101 is merged, I am about to release version 4.3.0-beta1 under the items-feed tag on npm to verify the functionality on your projects.

• I do expect to be done today

[Simply007]

Gatsby source plugin v.4.3.0-beta1 is released (under items-feed npm tag).

I have tested it out and I haven't spotted any issues.
@bwlng, @jupl - could you please try this version in your project to prevent any issues?

If no issues come out - v4.3 will be released on Thursday (Jan, 23rd).

[Simply007]

Version 4.3.0 is released 💪