User-agent: * Disallow: /files/ # this should prevent robots visiting any but the root directory