Hello @eddievet
Thank you for reaching out and I am happy to help!
In the recent 2.1.7 update, we fixed some issues with robots crawling the cached pages.
The problem is with the file permission so you should change the robots.txt permission to 644.
I hope this helps!
Thanks!
Thread Starter
EddieG
(@eddievet)
Hi
Problem solved. I created a robots.txt file and added the script you suggested
I didn’t have a robots.txt file in the first place although my website shows one under samplesite.com/robots.txt
I know that wp created one even though does not make it visible but i thought i would work, but not after creating the file it is all working as should.
Thanks for you time
p,s If you receive an email from me please ignore it
@vmarko I assume this fix introduced the problem stated in https://wordpress.org/support/topic/blocked-cache-directory-from-google-bots-triggering-mobile-friendliness-issue/ as well as a lot of other bugs.
Speaking from a bit more experience my advice would be to stay away from robots.txt unless you really know what you are doing. There is a reason why the robots.txt on bigger sites (even those with WordPress) often has a complexity of some hundreds of lines.
If you absolutely must touch it, you at least should allow users to customize easily what is added by your plugin and just give a default.
But that’s just my two cent and no complain since your plugin still does a great job with a fair pricing model 😉
Hello @eddievet
Thank you for the information.
@galfom yes you are correct. We are working on this and we already have a Github issue for this.
Please refer to the topic where the GH issue is posted:
https://wordpress.org/support/topic/w3tc-conflict-with-aioseo/
Thread Starter
EddieG
(@eddievet)
From what i know wp doesn’t display robots.txt on the wp root (public_html). However i curried on and applied robots.txt (now it is giving erron on rankmath plugin) and looks like things are back to normality (i will keep an eye on the development).
I need to know (based on others opinion) how robots.txt performs on other sites.
I’ve read many other complaints and i have to say that this error It cost me a lot of money.
I hope w3tc can fix this error once and for all.
Regards
Hello @eddievet @galfom
We have released a patch in version 2.1.8. We do apologize for any inconvenience. Please update and do let us know if there are any issues. We will be happy to assist you.
Thank you.
I updated W3TC only yesterday to 2.1.8 and the following robots.txt error appeared on the screen
FTP credentials don’t allow to write to file /var/www/xxxx/robots.txt
W3C Total Cache Error: Files and directories could not be automatically created to complete the installation.
Please execute commands manually or use FTP form to allow W3 Total Cache make it automatically.
Following is the content of robots.txt file
# BEGIN W3TC ROBOTS
User-agent: *
Disallow: /wp-content/cache/
# END W3TC ROBOTS
And most importantly, the plugin page becomes completely blank! Then, I had to go use the recovery mode and deactivate the plugin!
According to @vmarko I should have experienced this in the ealier version but most interestingly I never seen this messaage before updating to 2.1.8.
How can I get rid of this issue? I have a feeling that even 2.1.8 could not resolve the issue completely. Is this entirely an error coming from W3TC or other plugins could be responsible for this. If yes, could you figure out yet what might those be? Because this is very important to have the caching working properly!
Hello @subrataemfluence
Thank you for reaching out.
I would advise using the 2.1.6 version for the time being.
We are aware of the issues and this will be fixed in the upcoming 2.1.9 patch.
Thanks!
Thank you @vmarko for your suggestion! I will do the same
Hi Plugin Contributor the Problem not fixed in W3TC 2.1.8 Version . It again showing below conflict
# BEGIN W3TC ROBOTS
User-agent: *
Disallow: /wp-content/cache/
# END W3TC ROBOTS
Please resolve as soon as possible because it decreasing our organic Traffic . it is very bad update
Hello @freejobsalert3950
You can remove the rules manually and make sure that the option in Peformance>General settings>Miscellaneours>Enable robots.txt blocking for cache directory is disabled.
Thanks!
Hi @vmarko, as per your suggestion, I have downgraded the plugin from 2.1.8 to 2.1.6 and removed robots.txt successfully. No issue has been popped up since then.
Thank you!
