On March 9th we discovered and reported an issue to Yoast, the makers of the extremely popular Yoast SEO WordPress plugin. We found that their sitemap, which is automatically generated on every one of their 5+ million active installations, had been including links to attachments on password protected pages all the way up to their latest major version release, 7.0. Although 7.0 disables the attachment sitemap by default, the new version inherits the settings from previous versions, so all but newly configured sites running Yoast SEO would be affected.
How We Discovered The Issue
We noticed a strange pattern of referral links from someone in our live chat. Following them back, we saw that it was most likely a bot following links from our sitemaps. We noticed that one of those URLs was to a password protected post that the bot shouldn’t have been able to see. It turned out that the post itself hadn’t been accessed; just one of the attachments. In our case, it was a harmless screenshot but we were immediately concerned that this would affect others in much more significant ways.
How to Duplicate The Issue
Here’s what we found in a bit more detail. Let’s say you create a post and password protect it. The post itself isn’t included in the sitemap, but if you add an attachment, like an image or PDF file, that gets included in the attachment sitemap. This means that Google and other search engines will be told the location of confidential data on your WordPress site and encouraged to index it.
Furthermore, because the link to the attachment often includes the slug for the post itself, Yoast SEO is giving hackers a head start on finding private posts on your site where they can attempt to brute force the passwords. Here’s an example of an attachment sitemap:
As you can see, the URL not only includes the link to the attachment, but also the slug to the private post.
Yoast’s Response
Over two weeks have passed since we reported the bug. An issue (#9194) has been opened in Yoast’s Github tracker but it’s still under review and hasn’t been updated in almost two weeks. (Update 4/3/18: Version 7.2 includes a fix for this bug.)
Back in January, we wrote about a similar issue when we discovered the Divi WordPress theme was making password protected content public. We went public with the bug prior to Elegant Themes patching their code because resolving the issue would require more than a software update; webmasters would need to manually clean their indexed pages from search engines if they wanted their private data to be removed in a timely manner.
The bug with Yoast’s SEO plugin raises the same concern. It’s not enough to simply patch the software – though that is clearly needed – but webmasters also need to manually check to be sure password protected data hasn’t been indexed and if it has, request removal from the engines.
Next Steps
So, what’s next? We’ll update this post if we hear anything from Yoast and let you know when a fix is live, but for now here are the steps to take to get your private data removed from search engines.
- Review Your Site – Look around for posts/attachments that should be private but aren’t. Only posts can utilize the password protection feature, so if you’re unsure if you have any password protected posts just go to WP Admin->Posts and look for “– Password protected” beside the post title. Check to see if you have an attachment sitemap – www.domain.com/attachment-sitemap.xml – and inspect it for data attached to password protected posts.
- Check The Search Engines – Do some specific searches to see what’s been indexed. Search for exact matches. For example, if you have the words “Private Client Demo” in the title of your password protected post/attachment, search Google for “private client demo” and your domain name. Make a list of what you need to remove. More on how to check what content has been indexed here.
- Remove Private Content From Sitemaps – The latest version of Yoast, 7.x, allows you to disable attachment sitemaps altogether. Go to SEO>Search Appearance>Media and set “Redirect attachment URLs to the attachment itself?” to Yes.
- Manually Remove URLs from Search Engines – You can request any URL be removed from a search engine index. Here’s how to do it on Google and Bing.
Update 4/3/18: Version 7.2 of Yoast SEO, released today, includes a fix for this bug.
