Memory Fills Up Then A Buffer Error Is Generated.

I have SSIS sp2 running on a Win2003 64bit Server with 4processors and 16GB of ram. I am trying to load 1 billion rows of data into 10 tables. The source data is found in 12 different 50GB fixed width flat files stored on 2 different files servers. The destination is 10 different tables in a single SQL Server 2000 database which has 1TB of space allocated to it. I use the MS SQL OLE DB connection for each destination table.



The SSIS package is pretty straight forward. Everything takes place in 1 data flow. The 12 sources each flow through 12 different Row Count Transformations into a single Union All Transformation. From the Union All transformation the data goes into another Row Count Transformation then into a Conditional Split Tranformation. The data is split into 10 streams base on the last digit of one of the ID fields in the data. The 10 streams are fed to the 10 destination tables.



Every time I run the package (Start without Debugging) the avaible physical memory goes from around 15GB to 0 in about 2 minutes. The % comitted bytes in use goes from 5% to 100% in about 5 minutes. Once at 100% it will stay there for around 5 minutes before it will finally give me the following error message:



The system reports 98 percent memory load. There are 17178939392 bytes of physical memory with 189382656 bytes free. There are 8796092891136 bytes of virtual memory with 8742748930048 bytes free. The paging file has 54388109312 bytes with 16056320 bytes free.



This message is followed by a bunch of other messages:



SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Union All" (2073) failed with error code 0x8007000E. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.


SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread1" has exited with error code 0x8007000E. There may be error messages posted before this with more information on why the thread has exited.
The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.


SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Dr 2" (663) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.


...



SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Dr 3" (898) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.


SSIS Error Code DTS_E_THREADFAILED. Thread "SourceThread2" has exited with error code 0xC0047038. There may be error messages posted before this with more information on why the thread has exited.



I have tried adjusting the Engine threads down from 5 to 4 to 2. I have tried adjusting the FastLoadMaxInsertCommitSize from 1000000 to 100000 to 1000 (Destinations are tablocked and Check Constraints). I have tried moving the DefaultBufferMax up to 16500 and down to 2000.



Nothing works. The package fails everytime within 20 minutes of its start.



I would prefer not to have to rewrite the package and process each file sequentially as that would take forever.



Any ideas would be greatly appreciated.



Thanks.

-Scott