User-agent: * Sitemap: https://www.tds-int.com/sitemap.xml ############## # custom # ############## User-agent: * Disallow: /my/ Disallow: /website/info Disallow: /ar/website/info Disallow: /profile/ Disallow: /ar/profile/ Disallow: /forum Disallow: /ar/forum Disallow: /helpdesk/internal-requests-8 Disallow: /ar/helpdesk/internal-requests-8 Disallow: /ar/helpdesk/khdm-l-ml-1/knowledgebase Disallow: /helpdesk/odoo-support-1/forums Disallow: /ar/helpdesk/odoo-support-1/forums # Crawler policy (decided 2026-10-29): all public pages stay open to search engines # (Googlebot, Bingbot), AI search and answer crawlers (OAI-SearchBot, ChatGPT-User, # PerplexityBot, Claude-SearchBot, Claude-User, Applebot) and AI model-training # crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). No bot-specific groups are # used, so the rules above apply to every crawler. # The rules start with their own User-agent line (2026-11-06) so that strict # parsers, which end a group at a blank line, still apply the Disallow rules. # Login, signup and reset-password pages are NOT disallowed (2026-11-14): they carry # a noindex meta tag, and Google must be able to crawl them to see it. # No "Allow: /" line: everything not disallowed is allowed anyway, and parsers that # use the first matching rule would otherwise ignore every Disallow. User-agent: * Allow: /social_instagram/