# All robots will spider the domain User-agent: * Disallow: # Disallow directory /css User-agent: * # Disallow directory /vids User-agent: * # Disallow directory /downloadz User-agent: *