Create a robots.txt file in each folder mode project’s root, which will aggregate automatically in the default project’s public folder.
To enable this automatic aggregation, you need to set the GENERATE_AGGREGATED_ROBOTS_TXT environment variable to "true" for the container.
Setting this variable causes regeneration of the robots.txt on every container restart.
It is possible to prepend and append the aggregated robots.txt file by creating a robots.txt.pre for prepend and robots.txt.post for appending in the /projects/default/ folder.