An SEO professional was concerned about blocking critical CSS files from Google using robots.txt, such as *.css, php.ini, or .htaccess.
John explained that this sounds like a bad idea, even though the professional suggesteda few use cases.
For *.css – this would block all CSS files. Google has to be able to crawl CSS files so they can accurately render a site’s pages. If you block Google from crawling these files, they would not be able to assess critical information, such as whether the site is mobile-friendly.
Even though CSS files are not generally indexable on their own, Google still needs to see them.
For php.ini, this is a critical configuration file for PHP. It should already be locked down so that nobody can access it. That includes Googlebot.
For .htaccess, this is a special control file that cannot be accessed externally by default. Just like other lockdown files, it does not have to be explicitly disallowed from crawling.
John said that using someone else’s robots.txt file and assuming it will work is a dangerous strategy. Instead, you want to examine your site, think about which parts you don’t want crawled, and just disallow those in robots.txt.