Update documentation loader limit and optimize sitemap generation

The documentation loader has been updated to limit the number of items to 500 from the previous limit of 'Number.MAX_SAFE_INTEGER'. Additionally, the handling of URL generation and item retrieval in 'server-sitemap.xml/route.ts' has been refactored for more effective sitemap generation. A new 'robots.ts' file was created to set up default rules for web robots accessing the site.

apps/web/app/(marketing)/docs/_lib/server/docs.loader.ts

@@ -15,7 +15,7 @@ async function docsLoader(language: string | undefined) {
   const { items: pages } = await cms.getContentItems({
     collection: 'documentation',
     language,
-    limit: Number.MAX_SAFE_INTEGER,
+    limit: 500
   });
 
   return pages;