allowing spidering on protected pages

Discussion in 'Troubleshooting' started by ntc69, Apr 16, 2005.

  1. ntc69

    ntc69 Member

    Joined:
    Apr 9, 2005
    Messages:
    53
    I use an on-site search engine and was wondering how it would be possible to allow the associated spider to crawl my protected pages.

    I could just turn off protection for half an hour but is there a better way of doing it...using user-agents and the .htaccess file?
    ntc69, Apr 16, 2005
    #1
  2. alex

    alex aMember Pro Customer Staff Member

    Joined:
    Jan 24, 2004
    Messages:
    6,021
    alex, Apr 18, 2005
    #2

Share This Page