[engine] Show correct robots.txt on Tenant networks #4709

Ticket(s) / Related Merge Requests

• Minds MR: minds!217
• Helm MR: helm-charts!167 (merged)

Summary of Changes

Robots TXT change

Testing Considerations

This change is set up such that sandboxes will ALWAYS return the "deny all" robots file, thus full testing on sandbox isn't going to be possible.

Files:

• Permissive:

User-agent: *
Disallow: /api/
Allow: /api/v3/discovery
Allow: /api/v3/newsfeed/activity/og-image/
        
Sitemap: {$siteUrl}sitemap.xml

• Deny all:

User-agent: Twitterbot
Disallow:
User-agent: *
Disallow: /

Sandbox

• Check you get the deny all file when going to /robots.txt.

Staging

• /robots.txt on a tenant with a domain set has the permissive file.
• /robots.txt on minds.com has the permissive file.
• /robots.txt on a tenant subdomian (no domain setup) has the deny all file.

Deployment Considerations

Standard deployment.

Regression Scope

Changes robots.txt file to be engine driven.

Platform Affected (web, mobile, etc)

Web.

Developer Testing Completed

Manual, unit (phpspec)

Screenshots / Screen Recording

Does this impact

• Localization
• Dark/light mode
• Guest mode

Definition of Done Checklist

• The Acceptance Criteria has been met
• Code is tested: Testing includes unit/spec, E2E/automated and manual testing
• Merge requests description has been filled out