User-agent: * Disallow: /cgi-bin/ Disallow: /articles/ Disallow: /111/ Disallow: /scripts/ Disallow: /hold/ Disallow: /images/ Disallow: /fpimages/ Disallow: /imagesfp/ Disallow: /icons/ Disallow: /java/ Disallow: /presskit/ Disallow: /scimages/ Disallow: /timages/ Disallow: /sound/ Disallow: /video/ Disallow: /1list_fronpage.html Disallow: /email.html # Added 2/20/2007 - Kevin Rank Disallow: /cfinclude/ Disallow: /jscripts/ Disallow: /jscript/ Disallow: /junk/ Disallow: /temp/ Disallow: /temporary items/ Disallow: /tempsearch/ # Added 2/15/2008 - Kevin Rank - Trying something, exclude all, then include certain files. Did not work properly. Left the allow in, in case it works in future versions of robots.txt User-agent: * Allow: /about Disallow: /cgi-bin/ Disallow: /articles/ Disallow: /schedules/ Disallow: /111/ Disallow: /scripts/ Disallow: /hold/ Disallow: /images/ Disallow: /fpimages/ Disallow: /imagesfp/ Disallow: /icons/ Disallow: /java/ Disallow: /presskit/ Disallow: /scimages/ Disallow: /timages/ Disallow: /sound/ Disallow: /video/ Disallow: /1list_fronpage.html Disallow: /email.html # Added 2/20/2007 - Kevin Rank Disallow: /cfinclude/ Disallow: /jscripts/ Disallow: /jscript/ Disallow: /junk/ Disallow: /temp/ Disallow: /temporary items/ Disallow: /tempsearch/ # Added 2/15/2008 - Kevin Rank - Trying something, exclude all, then include certain files. Did not work properly. Left the allow in, in case it works in future versions of robots.txt Allow: /about Allow: /buildingbig Allow: /bythepeople Allow: /dialogue Allow: /dialogue4kids Allow: /dtv Allow: /dying Allow: /elections Allow: /evolution Allow: /hot Allow: /idahoaportrait Allow: /idahoarts Allow: /idahoportrait Allow: /idreports Allow: /kids Allow: /kidspbs Allow: /lc Allow: /learn Allow: /leglive Allow: /misunderstoodminds Allow: /mobile Allow: /news Allow: /ntti Allow: /outdoors Allow: /outreach Allow: /pbssites Allow: /productions Allow: /promotion Allow: /reachesout Allow: /salmonriver Allow: /schedules Allow: /specials Allow: /staycurious Allow: /support_new Allow: /takethisheart Allow: /zoom-action Allow: /*.cfm Allow: /*.html Allow: /*.htm # Added 5/30/2007 - Kevin Rank - Attempting to reduce spider traffic. # Crawl-Delay is a newish command that sets the ammount of time between pages being accessed by a spider. Time is in seconds. Usually, set between 0 - 10 #Crawl-delay: 10 Allow: /buildingbig Allow: /bythepeople Allow: /dialogue Allow: /dialogue4kids Allow: /dtv Allow: /dying Allow: /elections Allow: /evolution Allow: /hot Allow: /idahoaportrait Allow: /idahoarts Allow: /idahoportrait Allow: /idreports Allow: /kids Allow: /kidspbs Allow: /lc Allow: /learn Allow: /leglive Allow: /misunderstoodminds Allow: /mobile Allow: /news Allow: /ntti Allow: /outdoors Allow: /outreach Allow: /pbssites Allow: /productions Allow: /promotion Allow: /reachesout Allow: /salmonriver Allow: /schedules Allow: /specials Allow: /staycurious Allow: /support_new Allow: /takethisheart Allow: /zoom-action Allow: /*.cfm Allow: /*.html Allow: /*.htm # Added 5/30/2007 - Kevin Rank - Attempting to reduce spider traffic. # Crawl-Delay is a newish command that sets the ammount of time between pages being accessed by a spider. Time is in seconds. Usually, set between 0 - 10 #Crawl-delay: 3 # Added 10/30/2009 - Kevin Rank - For site search, enabled schedules to be crawled. Also reduced time from 10 to 3, in the above entry. Allow: /schedules/