Azure Library for Lucene.Net (Full Text Indexing for Azure)

"When Lucene asks to for a read stream for a segment (remember segments never change after being closed) AzureDirectory looks to see if it is in the local cache Directory, and if it is, simply returns the local cache stream for that segment. Otherwise it fetches the segment from blobstorage, stores it in the local cache Directory and then returns the local cache steram for that segment."

So my issue is everytime I rebuild and run the project local it needs to fetch all the segments needed for reading everytime which is time consuming in trying to develop the application.  What is a work around this issue?

If you mean you are running in azure emulator then just override the location with a temp folder name...something like:
 if(RoleEnvironment.IsEmulated)
      azureDirectory = new AzureDirectory(tempFolder);
 else
      azureDirectory = new AzureDirectory()

By default the temp folder which is used is the approved local storage temp folder for azure emulator, and every "deployment" in azure emulator creates a new local folder, which is why you have to download it each time.

I built an index using a winform app on my Windows 7 home addition. When I tried to use a Azure service to search from my local box, it works fine. But when it is deployed to azure, I am getting 

 [CorruptIndexException: doc counts differ for segment _1: fieldsReader shows 220 but segmentInfo shows 540]
   Lucene.Net.Index.FindSegmentsFile.Run() +1614
   Lucene.Net.Index.DirectoryIndexReader.Open(Directory directory, Boolean closeDirectory, IndexDeletionPolicy deletionPolicy) +68
   Maarg.EDIActive.WebRole.Controllers.SearchController.Index(String id, String

Any idea what I did wrong?

Thanks
Jianguo

Sounds like it didn't successfully commit blobs to storage?  I haven't seen this one

Hi. I've got a similar problem to what others are saying, but the responses just say that EOFs are part of normal operation, yet that's not what we're seeing...

We have tried to integrate Lucene.NET and AzureDirectory into our C# project on Azure. AzureDirectory and IndexSearcher are Singletons. And on "Azure Staging" it works great. But when we publish it to "Azure Production", we get a »read past EOF« exception. The AzureDirectory cache only contains 3 files out of 50, so all subsequent searches fail.


private static Lucene.Net.Util.Version LuceneVersion = Lucene.Net.Util.Version.LUCENE_30;
private static Microsoft.WindowsAzure.Storage.CloudStorageAccount cloudAccount = null;
private static AzureDirectory azureDirectory = null;
private static Lucene.Net.Search.IndexSearcher searcher = null;
private static StandardAnalyzer analyzer = null;

cloudAccount =
    new Microsoft.WindowsAzure.Storage.CloudStorageAccount(
        new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(System.Configuration.ConfigurationManager.AppSettings["LuceneAccount"],
                                                                   System.Configuration.ConfigurationManager.AppSettings["LuceneKey"]), true);

string filePath = RoleEnvironment.GetLocalResource("LuceneCache").RootPath;
azureDirectory = new AzureDirectory(cloudAccount, "mydbitems", new Lucene.Net.Store.SimpleFSDirectory(new DirectoryInfo(filePath)));

searcher = new IndexSearcher(azureDirectory, true);
analyzer = new Lucene.Net.Analysis.Standard.StandardAnalyzer(LuceneVersion);


Thanks for any ideas.
-Jerry-

I have 4 unique Lucene Indexes, basically different products for different countries. "lucenedb-us", "lucenedb-au", "lucenedb-eu", etc.

How exactly does the AzureDirectory caching work?

Will this:

azureDirectory = new AzureDirectory(cloudStorageAccount, "lucenedb-us", new RAMDirectory());

and then that:

azureDirectory = new AzureDirectory(cloudStorageAccount, "lucenedb-au", new RAMDirectory());

result in accurate results, or will the code always use "lucenedb-us"? If it will always use "lucenedb-us", will removing new RAMDirectory() fix the problem?

Create 4 azure directories, each with their own IndexSearcher and you will be good to go.  If you want to search across all of them then use a MultiSearcher() around the index searchers.

Please use https://github.com/richorama/AzureDirectory if you need a working version with Lucene 3 and Azure Tools 2.

no, it hasn't.  It has been moved to codeplex.  Bernhard, I wish you had contacted me about helping make the project better.

Not sure if you have run into this issue before or not.

I am using Lucene.Net (Version=3.0.3) and Lucene.Net.Store.Azure (Version=1.0.5.1) and get the following exception at runtime:

Declaration referenced in a method implementation cannot be a final method.  Type: 'Lucene.Net.Store.Azure.AzureDirectory'.  Assembly: 'Lucene.Net.Store.Azure, Version=1.0.5.1, Culture=neutral, PublicKeyToken=null'.

The following blog has the best write-up on what the meaning of the exception is: http://ayende.com/blog/1425/a-bug-story

Thanks a lot.

Please use https://github.com/richorama/AzureDirectory if you need a working version with Lucene 3 and Azure Tools 2.

We are developing a multitenant application, i would like to know which aproach i could use to handle very big catalog. Should i create a azuredirectory for each client? to have its index independent?


I possible to search encrypted data in lucene catalogs? thanks a lot.

I see you noted this change: " Replaced existing of blob lock file with blob leases to prevent orphaned lock files from happening"

However when I diff the code from my previous version (a few months old), the only difference I see is the remove of a Dispose in AzureDirectory.cs

Is that correct, or is something else missing?

Thanks!

Love the work,

Was wondering if you would object to me putting this up on GitHub and streamlining the whole nuget process from there etc...

It seems like the code will need some love once 3.0 releases and that is imminent.

It is high time for this to happen, but MSFT has a preference for Codeplex.  I have created a project on codeplex:
https://azuredirectory.codeplex.com/ 

Hello,
First of all, thanks for this, it's working great.  I'm wondering what the benefits would be of using this library versus mounting a cloud drive on all of the roles and using the normal filesystem storage in Lucene.  Is the caching/performance/cost better in your library than we would get with that approach?

Only one VHD can be writeable at a time.  With AzureDirectory, only 1 indexwriter can be operational at a time which is controllable by a lease blob, this means it is much easier to move around who is writing to the index at any given time.
Other than that, I can't really say how clouddrive would perform.