Server settings:
Windows Server 2003 standard edition SP1
SQL Server 2005 standard edition
MOSS 2007 standard edition SP1 (12.0.0.6219, find your version here http://www.mindsharpblogs.com/penny/articles/481.aspx)
The SharePoint server logged lots of event id 537 which is an audit failure. In this case it means that something is fishy in the connection between the crawl engine (run by the service account) and the site colletion.
Event ID 2436 and along with it this error message was logged:
http://intranet
Access is denied. Check that the Default Access Account has access to this content, or add a crawl rule to crawl this content.
(The item was deleted because it was either not found or the crawler was denied access to it)
In this case access is denied because the URL http://intranet is mapped to the loopback adapter of the SharePoint server and not because permissions are missing in the context. This means that the crawl engine is trying to connect to the loopback address (127.0.0.1) which isn’t allowed.
This a security feature in Windows Server 2003 SP1. If some of you run your Central Administration site and the crawl on separate servers you probably don’t get these issues.
The solution to this is to apply the registry fix described in MS KB896861 (http://support.microsoft.com/kb/896861)
Remember to back up the registry before making any changes. In this case, right-click the Lsa-folder and Export it to a location on the server or the network.
These are the steps outlined as method 2 in the KB article:
Disable the loopback check
1. Click Start, click Run, type regedit, and then click OK.
2. In Registry Editor, locate and then click the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. Right-click Lsa, point to New, and then click DWORD Value.
4. Type DisableLoopbackCheck, and then press ENTER.
5. -click DisableLoopbackCheck, and then click Modify.
6. In the Value data box, type 1, and then click OK.
7. Quit Registry Editor, and then restart your computer.