robots

robots makes meeko build write robots.txt with rules for User-agent: *: its allow and disallow entries (Allow: / if none are set) and, if sitemap is enabled, a Sitemap: line with its absolute URL. It works with all adapters: on Node, Bun, Deno and Workers it is served like any other public file, with an ETag.

Usage#

// meeko.config.ts
import { defineMeekoConfig } from "@meekojs/meeko/framework";

export default defineMeekoConfig({
  site: "https://example.com",
  sitemap: true,
  robots: { disallow: ["/admin/"] }, // or robots: true
});

Type#

readonly robots?: boolean | RobotsConfig

interface RobotsConfig {
	readonly allow?: readonly string[]
	readonly disallow?: readonly string[]
}

Options#

Each entry starts with /, with no spaces, ? or #. robots: true is equivalent to {}.

Result#

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

With robots: true and no sitemap:

User-agent: *
Allow: /

Examples#

A test environment that is not indexed#

import { defineMeekoConfig } from "@meekojs/meeko/framework";

const staging = process.env.DEPLOY_ENV === "staging";
export default defineMeekoConfig({
  robots: staging ? { disallow: ["/"] } : true,
});

Allowing a folder inside a disallowed one#

import { defineMeekoConfig } from "@meekojs/meeko/framework";

export default defineMeekoConfig({ robots: { disallow: ["/cuenta/"], allow: ["/cuenta/ayuda/"] } });

Common mistakes#

A misspelled field#

export default defineMeekoConfig({ robots: { disalow: ["/admin/"] } as never });

meeko dev, meeko build, meeko check and meeko routes stop with …/meeko.config.ts: robots: campo desconocido disalow.

import { defineMeekoConfig } from "@meekojs/meeko/framework";
export default defineMeekoConfig({ robots: { disallow: ["/admin/"] } });

A path without a leading /#

export default defineMeekoConfig({ robots: { disallow: ["admin"] } });

…/meeko.config.ts: robots.disallow debe ser un array de cadenas no vacías que empiecen por /.

import { defineMeekoConfig } from "@meekojs/meeko/framework";
export default defineMeekoConfig({ robots: { disallow: ["/admin"] } });

A public/robots.txt#

public/robots.txt   and   robots: true

meeko build stops with El archivo público robots.txt (de public/ o de un módulo) choca con el que genera la opción robots de meeko.config.ts; quita uno de los dos.

Delete public/robots.txt and declare its rules in robots

Notes#

• Not security. Disallow is a request to crawlers, not protection: protect /admin with a session and server middleware.
• Origin. robots without sitemap does not need site.
• Specific pages. To prevent a page from being indexed, use <meta name="robots" content="noindex"> with useSeoMeta; if it is prerendered, it is also excluded from the sitemap.

Related#

• sitemapWrites sitemap.xml during the build with prerendered pages, routes without parameters and, since 1.7, sitemapEntries URLs from dynamic routes.
• siteThe site's public URL, of which only the origin is used, for absolute URLs in sitemap.xml, robots.txt and document.seo.
• sitemap.xml and robots.txtMeeko as a full-stack framework