Allow crawler access to api

What does this MR do?

Related issue: #215105 (closed)

See #215105 (comment 395784968)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• [-] Documentation (if required)
• Code review guidelines
• [-] Merge request performance guidelines
• [-] Style guides
• [-] Database guides
• [-] Separation of EE specific content

Availability and Testing