Client: Improve SEO

Author: winwiz1

README.md

@@ -15,7 +15,16 @@
 </div>
 
 ## When To Use
-Crisp React offers innovative features to make your application more performant and potentially more secure. 
+Crisp React offers innovative features to make your application more performant and potentially more secure. The advantages do not come at the price of making your React code unportable which means that only a few stock components contain programming constructs specific to this solution.  All the components you write remain portable. There is no vendor lock-in.
+
+The solution provides full stack deployments with several cloud vendors yet again avoids vendor lock-in. This is complimented by Jamstack deployments that aim for simplicity and speedy production release. For all deployments, specific and verifiable (to achieve the stated goals) SEO instructions are provided.
+
+Sample websites:
+* [Demo - Full stack](https://crisp-react.winwiz1.com). Automated build performed by Heroku from this  [repository](https://github.com/winwiz1/crisp-react). All pages, including internal SPA pages, are noted in [sitemap](https://crisp-react.winwiz1.com/sitemap.xml) and [indexed](https://www.google.com/search?q=site%3Acrisp-react.winwiz1.com) by Google.
+* [Demo - Jamstack](https://jamstack.winwiz1.com). Automated build performed by Cloudflare Pages from the same repository. All pages, including internal SPA pages, are mentioned in [sitemap](https://jamstack.winwiz1.com/sitemap.xml) and [indexed](https://www.google.com/search?q=site%3Ajamstack.winwiz1.com) by Google.
+* [Production](https://virusquery.com). Based on Crisp React. All pages, including internal SPA pages, are noted in [sitemap](https://virusquery.com/sitemap.xml) and [indexed](https://www.google.com/search?q=site%3Avirusquery.com) by Google.
+
+The list demonstrates that Google can index SPA provided it is correctly written and deployed.
 
 [This section](docs/WhenToUse.md) answers the question “When use this solution” and also addresses the opposite e.g. “When not to use it”.
 
@@ -47,6 +56,7 @@ The section lists the features that are unique/innovative and those implemented
   - [Client Usage Scenarios](docs/Scenarios.md#client-usage-scenarios)
   - [Backend Usage Scenarios](docs/Scenarios.md#backend-usage-scenarios)
 - [Custom Domain and CDN](#custom-domain-and-cdn)
+- [SEO](#seo)
 - [What's Next](#whats-next)
 - [Pitfall Avoidance](#pitfall-avoidance)
 - [Q & A](#q--a)
@@ -224,11 +234,13 @@ The solution contains debuggable test cases written in TypeScript. It provides i
 
 The client and the backend can be tested independently by executing the `yarn test` command from their respective subdirectories. Alternatively the same command can be executed at the workspace level.
 
-The repository is integrated with [Travis CI](https://travis-ci.com) and [Heroku](https://heroku.com) for CI/CD and with [Cloudflare Pages](https://pages.cloudflare.com) for  CD. Every push to the repository causes Travis and Pages to start a VM, clone the repository and perform a build. Then Travis runs tests while Pages deploys the build to `xxxxx.crisp-react.pages.dev` and also makes it available on [crisp-react.pages.dev](https://crisp-react.pages.dev). This is followed by Heroku deployment, also automated and delayed until Travis tests finish successfully.
+The repository is integrated with [Travis CI](https://travis-ci.com) and [Heroku](https://heroku.com) for CI/CD and with [Cloudflare Pages](https://pages.cloudflare.com) for  CD. Every push to the repository causes Travis and Pages to start a VM, clone the repository and perform a build. Then Travis runs tests while Pages deploys the build to [jamstack.winwiz1.com](https://jamstack.winwiz1.com). This is followed by Heroku deployment, also automated and delayed until Travis tests finish successfully.
 
  The test outcome is reflected by the test badge and is also shown by the icon located after the last commit description and next to the last commit hash. To access more information, click on the icon. The icon is rendered as the check mark :heavy_check_mark: only if all the CI/CD activities performed by Travis CI, Heroku and Cloudflare Pages were successful. Otherwise an icon with the cross mark :x: is shown.
 ## Usage - Jamstack
-As already mentioned, you might prefer to simplify the Jamstack build by having one SPA called "index". This is achieved by having the SPA configuration block:
+Jamstack deployments do not use the Express backend. Static React files are served to clients by a server supplied by Jamstack provider. Therefore all Jamstack deployments are vendor-specific.
+
+As already mentioned, you might prefer to simplify deployments by having a single SPA called "index". This is achieved by having the following SPA configuration block:
 
 ```js
 /****************** Start single SPA Configuration ******************/
@@ -254,7 +266,9 @@ After the command finishes, the build artifacts are located in the `client/dist/
 Use the `yarn dev` and `yarn lint` commands executed from the `client/` directory to debug and lint Jamstack client.
 
 ### Cloudflare Pages
-Cloudflare Pages can build the client in the cloud, then create and deploy a website for it. This is done automatically provided the preparatory and configuration steps are completed:
+If you have the SPA configuration block as suggested at the beginning of this section, then a new website will be built and deployed to `*.pages.dev` domain after the steps listed below are completed. The follow-up [SEO](#seo) section is optional.
+
+If your SPA configuration block is different, then the newly built and deployed website will not work until the [SEO](#seo) section is completed.
 
 1. Clone Crisp React repository.
     ```
@@ -269,33 +283,37 @@ Cloudflare Pages can build the client in the cloud, then create and deploy a web
     ```
 4.  Deploy to Cloudflare Pages by logging into the [Cloudflare dashboard](https://dash.cloudflare.com). Use Menu > Pages > Create a project. You will be asked to authorize read-only access to your GitHub repositories with an option to narrow the access to specific repositories.
 
-    Select the repository which you pushed to GitHub at the previous step and in the "Set up builds and deployments" section, provide the following information:
+    Select the repository which you pushed to GitHub at the previous step and on the "Set up builds and deployments" screen, provide the following information:
     | Configuration option | Value |
     | :--- |:---|
     | Production branch  | `master`  |
     | Build command  | `yarn build:jamstack` |
-    | Build output directory  | `/client/dist`  |
+    | Build output directory  | `client/dist`  |
 
     Add the following environment variable:
 
     | Environment variable | Value |
     | :-------------------- | :----- |
     | NODE_VERSION | `16.13.0` |
 
-    Optionally, you can customize the "Project name" field. It defaults to the GitHub repository name, but it does not need to match. The "Project name" value is used to create a unique per-project `*.pages.dev` subdomain.
+    Optionally, you can customize the "Project name" field. It defaults to the GitHub repository name, but it does not need to match. The "Project name" is used to create a unique `*.pages.dev` subdomain. If the name is unique, it will be used as is, otherwise it will be altered a bit to ensure uniqueness. The resulting subdomain will be referred to as 'per-project subdomain' e.g. `<per-project>.pages.dev`.
+
+    After completing the configuration, click on the "Save and Deploy" button. You will see the deployment pipeline in progress. When it finishes, a website similar to [this](https://jamstack.winwiz1.com) can be found on the per-project subdomain. If there is no SPA named "index", you will have to navigate to "/your-spa-name".
 
-    After completing the configuration, click "Save and Deploy" button. You will see the deployment pipeline in progress. When it finishes, the website can be found on the above unique subdomain.
+Each subsequent push into the repository will trigger the pipeline. If it finishes successfully, a new website deployment is created and made available on both per-project and per-deployment subdomains. The latter comes in the form of `<per-deployment>.<per-project>.pages.dev`.
 
-Each subsequent push into the repository will trigger the pipeline. If it finishes successfully, a new website deployment is created and made available on both per-deployment URL and per-project subdomain. You can rollback to any of the earlier deployments anytime. Those are still available to users at the older per-deployment URLs. A rollback ensures that a website available at the created earlier per-deployment URL becomes accessible on the per-project subdomain as well.
+You can rollback to any of the previous deployments anytime. Those are still available to users at the older per-deployment subdomains. A rollback ensures that a website available at the created earlier per-deployment subdomain becomes accessible on the per-project subdomain as well.
 
-Getting the metrics (performance, SEO and others) of the new website is only few clicks away with the cloud instance of Lighthouse ran by Google and available at [this page](https://web.dev/measure/). The metrics should be similar to `crisp-react.pages.dev`:
+If you have the SPA configuration block as suggested at the beginning of this section, then getting metrics for the new website is only a few clicks away with the cloud instance of Lighthouse ran by Google and available at [this page](https://web.dev/measure/). The metrics should be similar to `jamstack.winwiz1.com`:
 
 <div align="center">
   <img alt="Jamstack build - Lighthouse scores" src="docs/benchmarks/jamstack.png" width="40%" />
 </div>
 
 > The report generated by web.dev has "CPU/Memory Power" metric at the bottom. It reflects the power of the hardware used by Lighthouse to emulate Moto G4. This metric affects the performance score. Cloud instance of Lighthouse at web.dev runs on a shared cloud VM and the metric reflects the current workload. It varies from time to time.
 
+In case the SPA configuration block is different, the metrics can be obtained after completing the [SEO](#seo) section.
+
 ### AWS S3
 Follow the steps described in the [AWS document](https://docs.aws.amazon.com/AmazonS3/latest/userguide/HostingWebsiteOnS3Setup.html).
 
@@ -306,6 +324,8 @@ Execute the build command shown at the beginning of this section and copy all th
 AWS CloudFront or Cloudflare CDN can optionally be used in front of the S3 bucket. In this case it's essential to ensure the CDN cannot by bypassed. The Stack Overflow [answer](https://stackoverflow.com/questions/47966890) shows how to restrict access to Cloudflare only.
 
 ## Usage - Full Stack
+All full stack deployments use the same Docker container to avoid vendor lock-in.
+
 Assuming the deployment demo in the [Project Highlights](#project-highlights) section has been completed, a container has already been built in the cloud and deployed to Google Cloud Run. In the current section we will build and run the container locally using Docker. This is followed by cloud deployments to Heroku and Google Compute Engine (GCE).
 ### Docker
 Install [Docker](https://docs.docker.com/get-docker/). To perform full stack build of Crisp React as a Docker image and start a container, execute Windows command file [`start-container.cmd`](https://github.com/winwiz1/crisp-react/blob/master/start-container.cmd) or Linux shell script [`start-container.sh`](https://github.com/winwiz1/crisp-react/blob/master/start-container.sh). Then point a browser to `localhost:3000`.
@@ -345,7 +365,7 @@ Perform the following steps after having cloned the repository:
     git remote set-url origin https://github.com/your-github-username/your-newly-created-repo
     git push
     ```
-3. Login to Heroku and create a new app. At this stage it has no content. Use the Settings tab to set the app's stack to 'container'. Then switch to the Deploy tab and choose GitHub as the deployment method. Finally trigger a manual build. Optionally enable automated builds.
+3. Login to Heroku and create a new app. At this stage it has no content. Use the Settings tab to set the app's stack to 'container'. Then switch to the Deploy tab and choose GitHub as the deployment method. Finally trigger a manual build. Optionally enable automated builds - this is how the [demo website](https://crisp-react.winwiz1.com) was deployed.
 
 #### Cloudflare CDN
 If you own a domain name, then it's recommended to add a CDN by implementing the optional steps described in the [Custom Domain and CDN](#custom-domain-and-cdn) section. It will significantly boost performance and improve security to some extent. The extent is limited due to the fact that the DNS record for your app e.g.`xxxxxx.herokudns.com` is public so the CDN can be bypassed with a potential attacker accessing your app directly.
@@ -479,7 +499,7 @@ This section complements the deployment described under the [Heroku](#heroku) he
 to take advantage of the distributed cache provided by Cloudflare and achieve much better performance with improved security. Both custom domain and CDN are optional. If you haven't used Cloudflare previously this [answer](https://www.quora.com/Cloudflare-product/How-does-Cloudflare-work-Does-Cloudflare-just-divert-malicious-traffic) could be useful.
 
 Prerequisites:
-- Custom domain name ownership,
+- Domain name ownership,
 - Cloudflare account. It's free and can be created by following this [link](https://dash.cloudflare.com/sign-up).
 
 The steps:
@@ -520,6 +540,8 @@ The steps:
 After the steps are completed the Heroku app will be using distributed caching and a free SSL certificate for the custom domain. Also the cache related statistics, monitoring and the breakdown of incoming requests by country will be available from Cloudflare even on the Free plan.
 
 Verify that integration with Cloudflare was successful by checking the page `https:/crisp-react.yourdomain.com/cdn-cgi/trace`. It should resemble the content of  `https:/crisp-react.winwiz1.com/cdn-cgi/trace`.
+## SEO
+Under :construction: construction
 ## What's Next
 Consider the following steps to add the desired functionality:
 ### Full stack build

client/src/entrypoints/first.tsx

@@ -26,12 +26,16 @@ import { isServer, getHistory } from "../utils/postprocess/misc";
 import "../css/app.css";
 import "../css/app.less";
 
+// If the first SPA is called 'first' then the regex
+// will match '/first' and '/first.html';
+const regexPath = new RegExp(`^/${SPAs.getRedirectName()}(\.html)?$`);
+
 const First: React.FC = _props => {
 
   const catchAll = () => {
     const path = window.location.pathname.toLowerCase();
 
-    if (path === ("/" + SPAs.getRedirectName())) {
+    if (regexPath.test(path)) {
       return <Overview/>
     }
 

client/src/utils/misc.ts

@@ -17,5 +17,6 @@ export const getCanonical = (pagePath?: string): string|undefined => {
 
 export const getTitle = (pageTitle?: string): string => {
   // eslint-disable-next-line no-extra-boolean-cast
-  return !!pageTitle? `${SPAs.appTitle} - ${pageTitle}` : SPAs.appTitle;
+  const ret = !!pageTitle? `${SPAs.appTitle} - ${pageTitle}` : SPAs.appTitle;
+  return ret + (CF_PAGES? " (Jamstack build)" : " (Full stack build)");
 }

client/src/utils/postprocess/postProcess.ts

@@ -86,22 +86,6 @@ Please check the 4-step sequence (provided in the comments at the top of each en
     console.log("Finished SSR post-processing")
   }
 
-  if (process.env.CF_PAGES) {
-    const writeFile = promisify(fs.writeFile);
-    const redirectName = require("../../../config/spa.config").getRedirectName();
-    const stapleName = "index";
-    const redirectFile = path.join(workDir, "_redirects");
-
-    if (redirectName.toLowerCase() !== stapleName) {
-      try {
-        await writeFile(redirectFile, `/ ${redirectName} 301`);
-      } catch (e) {
-        console.error(`Failed to create redirect file, exception: ${e}`);
-        process.exit(1);
-      }
-    }
-  }
-
   try {
     await postProcessCSS();
   } catch (e) {

deployments/cloudflare/worker-fullstack.js

@@ -0,0 +1,80 @@
+// Replace 'crisp-react.winwiz1.com' with the domain
+// you own. It will ensure that sitemap.xml contains
+// links specific to that domain (or subdomain).
+const siteMap = `<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+  <url>
+    <loc>https://crisp-react.winwiz1.com</loc>
+  </url>
+  <url>
+    <loc>https://crisp-react.winwiz1.com/a</loc>
+  </url>
+  <url>
+    <loc>https://crisp-react.winwiz1.com/namelookup</loc>
+  </url>
+  <url>
+    <loc>https://crisp-react.winwiz1.com/lighthouse</loc>
+  </url>
+  <url>
+    <loc>https://crisp-react.winwiz1.com/second</loc>
+  </url>
+</urlset>
+`;
+
+const bots = [
+  "googlebot",
+  "bingbot",
+  "yahoo",
+  "applebot",
+  "yandex",
+  "baidu",
+];
+
+class ElementHandler {
+  element(element) {
+    element?.replace('<div id="app-root"></div>', {html: true});
+  }
+
+  comments(comment) {
+  }
+
+  text(text) {
+  }
+}
+
+addEventListener("fetch", event => {
+  event.respondWith(handleRequest(event.request))
+})
+
+/**
+ * Respond to the request
+ * @param {Request} request
+ */
+async function handleRequest(req) {
+  const parsedUrl = new URL(req.url);
+  const path = parsedUrl.pathname.toLowerCase();
+  const lastIdx = path.lastIndexOf(".");
+  const extensionLess = lastIdx === -1;
+  const extension = path.substring(lastIdx);
+  const userAgent = (req.headers.get("User-Agent") || "")?.toLowerCase() ?? "";
+
+  
+  if (path === "/sitemap.xml") {
+    return new Response(
+      siteMap,
+      {
+        headers: {"content-type": "text/xml;charset=UTF-8"},
+      }
+    );
+  }
+  if ((extension === ".html" || extensionLess === true) &&
+    bots.some(bot => userAgent.indexOf(bot) !== -1)) {
+    const res = await fetch(req);
+    return new HTMLRewriter().on(
+	  "div[id='app-root']",
+	  new ElementHandler()
+	).transform(res);
+  } else {
+   return fetch(req);
+  }
+}