User-agent: ia_archiver Disallow: / User-agent: * #following applies to all disallow: /ar #exclude these subdirs disallow: /map disallow: /bin disallow: /cgi-local disallow: /etc disallow: /lib32 disallow: /stats disallow: release.html #exclude these particular files disallow: *.gif disallow: map.html disallow: /thework