An SEO professional asked John Mueller during a hangout about their robots.txt file. They were concerned about disallowed pages getting organic traffic.
They explained that they have disallowed some of the pages, but it is possible that Google had indexed these pages in the past. Although they disallowed crawling, to this day, they still see these pages getting organic sessions.
Why is this happening? And how can they fix that?
They asked about the noindex directive. But is this the right way to go about it?
John explained that if these are pages that you don’t want indexed, then using noindex is better than using the disallow in robots.txt.
The noindex would be a meta tag on the page. And you would need to allow crawling via robots.txt for noindex to physically happen.
This happens at approximately the 16:41 mark in the video.