Update documentation loader limit and optimize sitemap generation

The documentation loader has been updated to limit the number of items to 500 from the previous limit of 'Number.MAX_SAFE_INTEGER'. Additionally, the handling of URL generation and item retrieval in 'server-sitemap.xml/route.ts' has been refactored for more effective sitemap generation. A new 'robots.ts' file was created to set up default rules for web robots accessing the site.

apps/web/app/robots.ts

@@ -0,0 +1,13 @@
+import { MetadataRoute } from 'next';
+
+import appConfig from '~/config/app.config';
+
+export default function robots(): MetadataRoute.Robots {
+  return {
+    rules: {
+      userAgent: '*',
+      allow: '/',
+    },
+    sitemap: `${appConfig.url}/server-sitemap.xml`,
+  };
+}