Common Search Crawler GOTCHAS - Crawler complains "Access Denied" as WSS site has form authentication enabled

Many people experience the search not working on a SharePoint team with form authentication enabled. The common reason is that they do not have an extended web application (sharing the same content as the parent web application) on a different zone with NTLM/Windows Integrated authentication. The search crawler accesses to site content with NTLM account. It will get an Access Denied error when accessing to the form-authentication site. If you only use the windows Integrated authentication but still get one of the following errors, please make sure that your Content Access Account has enough (Full Read) privileges on the WSS sites. (Please see our "Grant Permission" section and select the web application using windows integrated authentication)

Error

• http://ragavtestsearch.com.au (Link does not work and is used for illustration purpose only)
• Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (The item was deleted because it was either not found or the crawler was denied access to it.)

Therefore, you will need a new web application with the same site content as the form-auth site and with the NTLM authentication enabled. How can you do it? You can use the SharePoint feature, "Extend an existing web application". Please follow the steps as below to extend a web application.

Extend the web application

1. Extend the form-authenticiation web application to a zone with windows-integrated authentication (NTLM) enabled
2. Go to Central Administration > Application Management page
3. Click Create or Extend web application and select Extend an existing web application
4. Make sure if the web application field shows the one you have set up the form authentication. If not, click on the drop down menu and select the web application in a popup dialog, e.g. SharePoint - 80
5. Either choose to use the randomly-generated port or assign a port. (The extended web application will be located at this port and share the same content as the original web application)
6. Select NTLM
7. Select No for Allow Anonymous option
8. Choose a zone (e.g. custom)
9. Click OK
10. After the web app is extended, the page will be redirected to the central administration page.
11. To test the site, please access to the extended web application and log on to the site with your search crawl acount.
    

Grant Permission

Please make sure if the search crawl account has access to the new web application. If not, please follow these steps to grant permission

1. Go to Central Administration > Application Management page
2. Click on Policy for web application
3. Click on Add Users
4. Make sure if the web application field shows the one you have set up the form authentication.
5. Select the zone that you extended the web application (e.g. custom)
6. Click Next
7. Enter the crawl account username to Users field
8. Check Full Read for the permission
9. Click Finish
10. To test the site, please access to the extended web application and log on to the site with your search crawl acount.
    

In the next crawl, the crawler will access to the site content through this “secret” NTLM web application. It will be able to get the form authentication descriptor of the content. After the crawl’s completed, you will see search results in your form-authentication site. It’s important to know that the URL of search hit will be based on your form-authentication site, because it’s depending on where you are searching from.

(Thanks to the great tips offered by wsssearch website. You can find more information about this website here)