Azure Library for Lucene.Net (Full Text Indexing for Azure)

I built an index using a winform app on my Windows 7 home addition. When I tried to use a Azure service to search from my local box, it works fine. But when it is deployed to azure, I am getting 

 [CorruptIndexException: doc counts differ for segment _1: fieldsReader shows 220 but segmentInfo shows 540]
   Lucene.Net.Index.FindSegmentsFile.Run() +1614
   Lucene.Net.Index.DirectoryIndexReader.Open(Directory directory, Boolean closeDirectory, IndexDeletionPolicy deletionPolicy) +68
   Maarg.EDIActive.WebRole.Controllers.SearchController.Index(String id, String

Any idea what I did wrong?

Thanks
Jianguo

Sounds like it didn't successfully commit blobs to storage?  I haven't seen this one

Hi. I've got a similar problem to what others are saying, but the responses just say that EOFs are part of normal operation, yet that's not what we're seeing...

We have tried to integrate Lucene.NET and AzureDirectory into our C# project on Azure. AzureDirectory and IndexSearcher are Singletons. And on "Azure Staging" it works great. But when we publish it to "Azure Production", we get a »read past EOF« exception. The AzureDirectory cache only contains 3 files out of 50, so all subsequent searches fail.


private static Lucene.Net.Util.Version LuceneVersion = Lucene.Net.Util.Version.LUCENE_30;
private static Microsoft.WindowsAzure.Storage.CloudStorageAccount cloudAccount = null;
private static AzureDirectory azureDirectory = null;
private static Lucene.Net.Search.IndexSearcher searcher = null;
private static StandardAnalyzer analyzer = null;

cloudAccount =
    new Microsoft.WindowsAzure.Storage.CloudStorageAccount(
        new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(System.Configuration.ConfigurationManager.AppSettings["LuceneAccount"],
                                                                   System.Configuration.ConfigurationManager.AppSettings["LuceneKey"]), true);

string filePath = RoleEnvironment.GetLocalResource("LuceneCache").RootPath;
azureDirectory = new AzureDirectory(cloudAccount, "mydbitems", new Lucene.Net.Store.SimpleFSDirectory(new DirectoryInfo(filePath)));

searcher = new IndexSearcher(azureDirectory, true);
analyzer = new Lucene.Net.Analysis.Standard.StandardAnalyzer(LuceneVersion);


Thanks for any ideas.
-Jerry-

I have 4 unique Lucene Indexes, basically different products for different countries. "lucenedb-us", "lucenedb-au", "lucenedb-eu", etc.

How exactly does the AzureDirectory caching work?

Will this:

azureDirectory = new AzureDirectory(cloudStorageAccount, "lucenedb-us", new RAMDirectory());

and then that:

azureDirectory = new AzureDirectory(cloudStorageAccount, "lucenedb-au", new RAMDirectory());

result in accurate results, or will the code always use "lucenedb-us"? If it will always use "lucenedb-us", will removing new RAMDirectory() fix the problem?

Create 4 azure directories, each with their own IndexSearcher and you will be good to go.  If you want to search across all of them then use a MultiSearcher() around the index searchers.

Please use https://github.com/richorama/AzureDirectory if you need a working version with Lucene 3 and Azure Tools 2.

no, it hasn't.  It has been moved to codeplex.  Bernhard, I wish you had contacted me about helping make the project better.

Not sure if you have run into this issue before or not.

I am using Lucene.Net (Version=3.0.3) and Lucene.Net.Store.Azure (Version=1.0.5.1) and get the following exception at runtime:

Declaration referenced in a method implementation cannot be a final method.  Type: 'Lucene.Net.Store.Azure.AzureDirectory'.  Assembly: 'Lucene.Net.Store.Azure, Version=1.0.5.1, Culture=neutral, PublicKeyToken=null'.

The following blog has the best write-up on what the meaning of the exception is: http://ayende.com/blog/1425/a-bug-story

Thanks a lot.

Please use https://github.com/richorama/AzureDirectory if you need a working version with Lucene 3 and Azure Tools 2.

We are developing a multitenant application, i would like to know which aproach i could use to handle very big catalog. Should i create a azuredirectory for each client? to have its index independent?


I possible to search encrypted data in lucene catalogs? thanks a lot.

I see you noted this change: " Replaced existing of blob lock file with blob leases to prevent orphaned lock files from happening"

However when I diff the code from my previous version (a few months old), the only difference I see is the remove of a Dispose in AzureDirectory.cs

Is that correct, or is something else missing?

Thanks!

Love the work,

Was wondering if you would object to me putting this up on GitHub and streamlining the whole nuget process from there etc...

It seems like the code will need some love once 3.0 releases and that is imminent.

It is high time for this to happen, but MSFT has a preference for Codeplex.  I have created a project on codeplex:
https://azuredirectory.codeplex.com/ 

Hello,
First of all, thanks for this, it's working great.  I'm wondering what the benefits would be of using this library versus mounting a cloud drive on all of the roles and using the normal filesystem storage in Lucene.  Is the caching/performance/cost better in your library than we would get with that approach?

Only one VHD can be writeable at a time.  With AzureDirectory, only 1 indexwriter can be operational at a time which is controllable by a lease blob, this means it is much easier to move around who is writing to the index at any given time.
Other than that, I can't really say how clouddrive would perform. 

Help

I followed your example, although used Lucene 2.9.4 and .NET 4.0.  When creating the index on Azure it goes fine.  When searching and creating an instance of the index searcher, it it throws an error :  File not found segments_5t.  What is going on?  I can see that the indexing has NOT created this file.

The file list I can see on azure after indexing is:

_1.fdt
_1.fdx
_1.fnm
_1.frq
_1.nrm
_1.prx
_1.tii
_1.tis
segments.gen
segments_2

Please help

as mentioned twice before, Lucene.Net uses File not found and read past EOF exceptions as part of normal operations. Changed your debugger to not stop on those exceptions.