Flat File Data Reader

NEW POST: IDataReader Interface Implementation

dbrowne — Wed, 02 Apr 2008 15:47:45 GMT

I uploaded a new version today. Implemented IDataReader.GetSchemaTable, which is required to load a DataTable. There's some new sample code in the program showing loading the flat file into a DataTable.

Thanks for the input,

David

UPDATED RELEASE: FileLoader.zip (Feb 18, 2008)

Wed, 02 Apr 2008 15:40:52 GMT

This zip has the FlatFileDataReader.cs and a sample project.

UPDATED RELEASE: Flat File Data Reader (Apr 02, 2008)

Wed, 02 Apr 2008 15:37:24 GMT

This release adds a simple implementation of IDataReader.GetSchemaTable, which is necessary to be able to load a DataTable from an IDataReader.

CREATED RELEASE: Flat File Data Reader (Apr 02, 2008)

Wed, 02 Apr 2008 15:35:05 GMT

This release adds a simple implementation of IDataReader.GetSchemaTable, which is necessary to be able to load a DataTable from an IDataReader.

NEW POST: IDataReader Interface Implementation

DaveEvans — Wed, 26 Mar 2008 19:12:15 GMT

DataTable.Load Compatibility

Thanks for this nice little problem solver you came up with here. I have several files ranging in size from 2k to 100MB. I want to use the same tool to load them all (a datareader). I wanted to load to smaller files into a DataTable using the DataTable.Load(IDataReader) method. Currently this method returns no rows. Whilst I understand that this reader was designed for large files and therefore loading results into a DataTable would be ‘nuts’ is this a simple fix/change?

NEW POST: IDataReader Interface Implementation

DaveEvans — Wed, 26 Mar 2008 19:11:44 GMT

Discussing the implentation of the IDataReader

UPDATED WIKI: Home

dbrowne — Mon, 03 Mar 2008 17:07:23 GMT

Resource Page Description
An abstract IDataReader implementation that reads flat files. Enables bulk loading of SQL Server using SqlBulkCopy from flat files with strange formats.

For ordinary delimited or fixed-width files, SSIS is a better tool, but when your file format doesn't work well with these tools or you need a pure .NET solution, consider using this FlatFileDataReader. Includes the option to skip over lines in the file for multi-format files.

Good scenarios are loading from clients (without SSIS), and loading files with multiple record layouts, heders and footers, etc.

Package includes a sample solution.

The FlatFileDataReader base class will process the file line-by-line. Each line will be passed to an abstract class member for you to parse out the column values and store them in local variables.

Here are the abstract members that you would need to implement to read your file:

 
        /**
         * 
         * Implement this to receive each line in the file;
         * After each row is read from the file, this method
         * will be fired.
         * 
         * Return END_OF_FILE if this is the trailer record
         * or to otherwise signal an End-of-file.
         * 
         * Return SKIP if this row is just ignored.
         * 
         * Return READ if this is a valid row
         * 
         * Optionally alter the line (it's passed as a ref).
         * 
         * Throw an exception if the line is invalid.
         * 
         */
        public abstract LineResult ReadLine(ref string line);
 
        /**
         * Readers will call this to get a value
         * Return value should be the proper .NET type for the field         * 
         */
        public abstract object GetValue(int i);
 
        /**
         * These methods return metadata about the fields
         * */
        public abstract Type GetFieldType(int i);
        public abstract string GetName(int i);
        public abstract int FieldCount
        {
            get;
        }

Use the Issue Tracker tab for problems or feature requests, and let me know if you find this sample useful.

David

UPDATED WIKI: Home

dbrowne — Mon, 03 Mar 2008 17:06:24 GMT

 
        /**
         * 
         * Implement this to receive each line in the file;
         * After each row is read from the file, this method
         * will be fired.
         * 
         * Return END_OF_FILE if this is the trailer record
         * or to otherwise signal an End-of-file.
         * 
         * Return SKIP if this row is just ignored.
         * 
         * Return READ if this is a valid row
         * 
         * Optionally alter the line (it's passed as a ref).
         * 
         * Throw an exception if the line is invalid.
         * 
         */
        public abstract LineResult ReadLine(ref string line);
 
        /**
         * Readers will call this to get a value
         * Return value should be the proper .NET type for the field         * 
         */
        public abstract object GetValue(int i);
 
        /**
         * These methods return metadata about the fields
         * */
        public abstract Type GetFieldType(int i);
        public abstract string GetName(int i);
        public abstract int FieldCount
        {
            get;
        }

CREATED RELEASE: FileLoader.zip (Feb 18, 2008)

Mon, 18 Feb 2008 18:35:20 GMT

This zip has the FlatFileDataReader.cs and a sample project.

UPDATED WIKI: Home

dbrowne — Mon, 18 Feb 2008 18:33:15 GMT