We are trying to load flat text files with upwards of 7 million records into a table on SQL. The table has a clustered index on 3 fields. We are sometimes able to complete smaller tables (500,000-750,000 records) and build the indexes prior to importing the data, however when we try the larger tables an error occurs :
Error at Destination for row number 6785496. Errors encountered so far in this task: 1
Location: somerge.c:1573
Expression: mrP->mrStatus!=MERGERUN::NONE
SPID: 11
Process ID: 173
None of the recods end up importing. The row number it gives is always the total number of records that was in the text file I was trying to import. I tried to import the text files first and then build the clustered indexes but a table with only 300,000 records ran for nearly 4 days without completing before we killed it.
I have got an xml file with size more than 2 GB. I have to load this file into tables. With 32 bit platform, I am unable to load this file using SSIS. Ram is 8 GB, but it is still bombing out. As I know it uses XML DOM Parser and tries to shred the file in memory and because of memory limition, it fails. Although I have already written code in C# using XmlTextReading object(implemetation of SAX Parser) to load data in tables, but I want to keep this loading process within the limits of DBAs.
I am stuck. Can someone guide me through the situation?
I'm having a very irritating time trying to migrate data from a COBOL system to SQL Server.
One of the A/R Master files has approx. 200 columns.
I can export this file any number of ways that will (sometimes) load partially into my database, but always when the load succeeds, columns 75 through N simply contain NULL, even though there is data in the file. When the load fails in DTS, the error is always missing column delimiter. Using BULK INSERT the error is always something like data too long at column 75.
Putting all this together, I have deduced that something isn't working if I try to load a staging table with more than 74 columns of data. This seems like a way-too-low threshold for a robust database, especially since SQL Server is supposed to be able to handle up to 1,024 columns per table.
I am having trouble loading tables (within the same data flow) that have a foriegn key relationship defined between them. For instance:
Table A is a parent (one side of the relationship) to Table B (many side of the relationship).
I am trying to load Table A first within the data flow and then Table B after, but I get the following error:
[OCMD EntityRole Insert [2666]] Error: An OLE DB error has occurred. Error code: 0x80040E2F. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E2F Description: "The statement has been terminated.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E2F Description: "The INSERT statement conflicted with the FOREIGN KEY constraint "FK_EntityRole_Entity". The conflict occurred in database "ODS", table "dbo.Entity", column 'EntityGuid'.".
I am currently using OLE DB commands to perform the inserts, I load table A and move on to then load Table B, I can see the records in Table A before trying to load Table B but for some reason Table B load still fails.
I was thinking maybe this has something to do with the transaction setting or Isolation level but have played with this to no avail (currently everything is the default - supported/serializable). Also, I am thinking maybe because the OLE DB commands are creating two seperate connections (they are using the same connection manager) the second one is unable to see the transactions from the other (first) connection (Table A)?
Is there a way around this without dropping (disabling) forigen keys before the load and adding them back in after? Would like to avoid this?
I would also like to avoid reading the data source multiple times. Everything I need is in the one source so I would like to populate multiple tables from the one source data stream instead of reading the same data 2,3 or 4 times etc.
Seems to me there must be a simple explanation/solution for this but I'm stuck at this point?
P.S.
I was intially using OLE DB destinations (because they are much faster) and was having the same issue, which made sense because the OLE DB destinations do not let you pass the data stream on so I had to multi cast to the destinations so they were loading at the same time. I would rather use the OLE DB destinations so if you have any ideas around how I could do this using those components that would be appreciated too!
I am trying to do here is to load different flat files to different tables: For example, if the file name starts with "enrollment", then it goes to table "enrollment" table; if the file name starts with "student", then it goes to "student" table.
For now, I created a foreach loop container for the each different files. So it ended up using several foreach loop containers. I am wondering if there is a way just to use one foreach loop containters to process the loading.
Anyone shed some light on this?? Thank you very much for your help!
Hi I have a question how to load data to tables linked by Foreign Keys in MSDE/SQL server. Example: If I have 2 tables linked (by Foreign Key): One table:
ITEM idITEM NAMEITEM CATEGORY (FK) 1cheese 2
And another:
Category IDCATEGORY NAME 1 household 2 food 3 general
How do I enter the load of data Do I have to enter it as 1cheese2 or is there some way of entering it as 1cheesefood
TDS wizard does not allow me to transfer to views/querries what I thought would be a normal way as I would enter data to view(relevant to Access's form) and it would update related tables . When I wrote sql to do it it said I can not update my view table as too many tables would be affected(I had lookup tables empty then though) I am doing it by number using TDS wizard to transfer it directly to the main table but there must be a better way
Is there a way ,(if so what is the syntax?), to set up a DTS package that loads a table that has an identity column. I am trying to load the data from another table, (leaving the identity field unmapped), and de-selecting the "enable identity insert" from the advanced tab of the Data Transformation Properties window. I keep getting errors due to the table not allowing null values. I tried using the set_identity command, but this still did not work. Any help would be appreciated. TB
Currently we are trying to load the xml files into sqlserver tables by using ssis 2012,We are getting xml files as a column in source table ,so we have to push these xml files into destination tables.
I'm following the below way to perform this activity
[URL]
But We have standard XSD structure for all the xml files ,and if xml file matches the XSD structure then only we have to load ,else it should skip to next xml file.
The reason it is poorly designed is the table is used to hold questions and answers, all with a 1:1 relationship. Instead of having ID, ProductType, Question, Answer they have unfortunately adopted the approach of the above i.e:
id 1 thisID 3 parentid nuLL DESCRIPTION: this is a question
id 20 thisID 3_1 parentID 3 DESCRIPTION: this is the answer to the question above
So I am writing a sproc that does this using a temp table. I got this far:
SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO -- ============================================= -- Author:Spencer -- ============================================= ALTER PROCEDURE [dbo].[GetFAQs] -- Add the parameters for the stored procedure here @ProductType varchar(255) AS BEGIN SET NOCOUNT ON; -- Insert statements for procedure here CREATE TABLE TEMP ( IDINT, thisIDvarchar(50) null, parentIDvarchar(50) null, Titlevarchar(255) null, DescriptionQvarchar(8000) null, DescriptionAvarchar(8000) null, ProductTypevarchar(255) null, )
SELECT ID, thisID, parentID, Title, DescriptionQ, DescriptionA, ProductType FROM A2Z WHERE ProductType = @ProductType AND parentID IS NULL END GO
This gets all my questions for that product type.
What I need to do is load the questions into my temp table and then run through the a2z table again gaining the answers to the questions (the parentid holds the question ID). The answers then will also get loaded into the temp table.
I have problem in loading multiple excel sheets data in to according to that excelsheets tables in a DB. All the excel sheets are in a folder,from that folder i have to acces all excel sheets. For this i am unsing script task and one dataflow task. But the error is coming in script task i am not able to put the path in the script..
Is this the correct way to do like this? Or any other way?
Can u please tell me the solution for this..Thanks in advance who are responding to this mail...
I am facing a peculier problem. Problem definition goes like this,
I have one staging DB in which all the tables resides in Primary file and one production DB in which tables resides in 2 secondary files.
Now when iam trying to load the data from the table A in staging which is on primary file to the table A1 in production DB which in secondary file, all the data are going to error log instead of table A1.
when i use the query designer and add tables it takes 15-30 mins to load a table....so if i need 6 tables in my query thats like 6 times 30 minute load times...i'm using oracle as my data source
when i switch the source to oledb it adds tables within seconds the only thing is i need to use named parameters
my dba folks are coming back with an answer to a question that sounds strange and I thought I would check on here.
Situation.
We are using an ETL tool I developed to move data around and building a DW on 2005. In some cases we are using the bulk loader to load and in some cases the tool itself.
One of the defaults of the tool is to truncate trailing blanks....and so fields that contain only blanks get truncated to a zero length character string.
When reloading the data with the tool it carries a null indicator next to the field value so the zero length character string is loaded as not null.
When loading with the bulk loader the dbas are telling me that the field is translated to a null. Note that they want some fields translated to null so they are using the keepnulls parameter.
On other databases (and built into the tool because this is so) the bulk loaders usually allow the specification of the load statement at column level and the specification of a 'null character sting' to be translated to null if this string is found. I put an example below.
I seem to recall that SQL Server 7 had some sort of bulk loader that allowed specification of columns at column level......for example offsets or whatever and the specification of fields to be interpreted as nulls. (Though that was a long time ago.)
I have searched through the manual and I don't see an option there any more to specify a character that will be interpreted to a null by the bulk loader.
Is it possible in 2005 to specify a character such that when the bulk loader sees it the field will be set to null?? And not just set fields to null which are not present in the load file?
(Just by the way, we are going to make the truncate trailing blanks optional and it's easy....it's just that I thought this kind of null if option was available in 2005 and I am keen to know if it is not there, either gone or never was...)
Thanks in Advance and Best Regards
Peter
Example of how Oracle does it...on page
http://www.csee.umbc.edu/help/oracle8/server.815/a67792/ch05.htm#5754 NULLIF Keyword Use the NULLIF keyword after the datatype and optional delimiter specification, followed by a condition. The condition has the same format as that specified for a WHEN clause. The column's value is set to null if the condition is true. Otherwise, the value remains unchanged. NULLIF field_condition
The NULLIF clause may refer to the column that contains it, as in the following example: COLUMN1 POSITION(11:17) CHAR NULLIF (COLUMN1 = "unknown")
I am trying load data from multiple Foxpro tables which are under a folder. I can have multiple folders with 17 foxpro tables. I was able to do it in DTS using ActiveX script. Here is the ACtiveX script.
'********************************************************************** ' Visual Basic ActiveX Script '************************************************************************ Option Explicit Dim conObj,DSNGosfbill,comObj,objRs,HostServer Dim sFolder,sFileFolder, Details,subFolderoccur,sFileFolderDBF,sFileFolderFPT,CheckFile,dFiles,Fil Dim fso, folderObj,subFolderList,dFolderObj Dim objPackage,oStep,objPackage_1,oStep_1,ConnObj_001,ConnObj_004,ConnObj_031,ConnObj_032,ConnObj_033 Dim ConnObj_Hclaimb, ConnObj_HProv, ConnObj_Hids, ConnObj_HCodes, ConnObj_HSpan, ConnObj_002, ConnObj_HCHGB Set conObj = CreateObject("ADODB.Connection") HostServer =DTSGlobalVariables("gvServer").Value
set comObj=CreateObject ("adodb.command") set comObj.ActiveConnection =conObj Function Main() Dim Dir_Name,DirFlag Dir_Name = "" DirFlag = "N" Set fso = CreateObject("Scripting.FileSystemObject")
If fso.FileExists(checkFile) Then Else Details = "***** Success.Lst file is missing in Batch folder. BATCH job may not be successfull or there are no folders in UNZIP directory to process. Check the batch run.*****" Call Write_Log Main = DTSTaskExecResult_Failure Exit Function End If Set folderObj = fso.GetFolder(sFolder) Set subFolderList = folderObj.SubFolders For Each subFolderOccur in subFolderList DirFlag = "Y" Dir_Name = subFolderOccur.Name Call Process_Dir(1,subFolderOccur.Name) Next If DirFlag = "N" Then Details = "***** No directories to process in SSI UNZIP folder*****" Call Write_Log End If If DirFlag = "Y" Then Call Process_Dir(2,Dir_Name) If objRs.Eof Then Details = "***** No directories to process in SSI UNZIP folder*****" Call Write_Log End If While not objRs.EOF set sFileFolder = fso.GetFolder(sFolder & objRs("zip_file_name")) Details = "***** Start-Time " & sFileFolder & " " & Date & " " & Time & "*****" Call Write_Log Call Update_Process_Flag("L",objRs("zip_file_name")) '*******Execute the package for each directory****************' '********* Call the Package**************' Set objPackage = CreateObject("DTS.Package") Set objPackage_1 = CreateObject("DTS.Package")
Set ConnObj_001 = objPackage.Connections("SSIPATH001") ConnObj_001.DataSource = sFileFolder
Set ConnObj_002 = objPackage.Connections("SSIPATH002") ConnObj_002.DataSource = sFileFolder
Set ConnObj_004 = objPackage.Connections("SSIPATH004") ConnObj_004.DataSource = sFileFolder Set ConnObj_031 = objPackage.Connections("SSIPATH031") ConnObj_031.DataSource = sFileFolder Set ConnObj_032 = objPackage.Connections("SSIPATH032") ConnObj_032.DataSource = sFileFolder Set ConnObj_033 = objPackage.Connections("SSIPATH033") ConnObj_033.DataSource = sFileFolder
Set ConnObj_Hclaimb = objPackage.Connections("SSIPATHCLAIMB") ConnObj_Hclaimb.DataSource = sFileFolder
Set ConnObj_HProv = objPackage.Connections("SSIPATHPROV") ConnObj_HProv.DataSource = sFileFolder Set ConnObj_Hids = objPackage.Connections("SSIPATHHIDS") ConnObj_Hids.DataSource = sFileFolder Set ConnObj_HCodes = objPackage.Connections("SSIPATHCODES") ConnObj_HCodes.DataSource = sFileFolder Set ConnObj_HSpan = objPackage.Connections("SSIPATHSPAN") ConnObj_HSpan.DataSource = sFileFolder
Set ConnObj_HCHGB = objPackage.Connections("SSIPATHCHGB") ConnObj_HCHGB.DataSource = sFileFolder
objPackage.Execute For Each oStep In objPackage.Steps If oStep.ExecutionResult = DTSStepExecResult_Failure Then Details = "***** GOSFBILL_SSI_Staging_Load failed. " & Date & " " & Time & "*****" Call Write_Log Main = DTSTaskExecResult_Failure Exit Function End If Next
For Each oStep_1 In objPackage_1.Steps If oStep_1.ExecutionResult = DTSStepExecResult_Failure Then
Details = "***** GOSFBILL_SSI_Update_FileSource failed. " & Date & " " & Time & "*****" Call Write_Log Main = DTSTaskExecResult_Failure Exit Function End If Next
'********************************************' Details = "***** End-Time " & sFileFolder & " " & Date & " " & Time & "*****" Call Write_Log objPackage.Uninitialize objPackage_1.Uninitialize Set objPackage = Nothing Set objPackage_1 = Nothing sFileFolder = "" sFileFolderDBF = "" sFileFolderFPT = "" objRs.MoveNext Wend objRs.Close End If Call Close_Conn Main = DTSTaskExecResult_Success End Function Sub Process_Dir (Para_cntl,Dir_Name) comObj.CommandText ="dbo.Usp_Process_Dir" comObj.commandtype = 4 comobj.parameters.Refresh comobj.parameters("@Para_Cntl")= para_cntl comobj.parameters("@Dir_Nm")= Dir_Name comobj.parameters("@File_Type")= "SSI" If (Para_Cntl = 1)Then comObj.Execute() Else If Para_Cntl = 2 Then Set objRs = comObj.Execute() End If End If
End Sub Sub Update_Process_Flag(P_Flag,Dir_Name) comObj.CommandText ="dbo.Usp_Process_Flag" comObj.commandtype = 4 comObj.parameters.Refresh comObj.parameters("@Process_Flag")= P_Flag comobj.parameters("@Dir_Nm")= Dir_Name comObj.Execute() End Sub Sub Write_Log comObj.CommandText ="dbo.usp_etl_write_log" comObj.commandtype = 4 comobj.parameters.Refresh comobj.parameters("@Text")= Details Comobj.parameters("@NDC_SSI_IND")= "SSI" Comobj.parameters("@Process_Stage")= "Staging" comObj.Execute() End Sub
Sub Close_Conn Set comObj = Nothing Set objRs = Nothing conObj.Close Set conObj = Nothing Set fso = Nothing Set folderObj = Nothing Set subFolderList = Nothing End Sub
When I migrated this code to SSIS, its not working. How can I achive this functionality in SSIS. Any one pls help me.
I have Table A . we already have 80 columns . we have to add 65 more columns.
we are populating this table from oracle .and we need to populate those 65 columns again from the same table.
Is it a better idea to add those new 65 columns to the same table or new table.
If we go for the same table then loading time will be double, If I go for new table and If i am able to run both the packages which loads table data from same oracle server to difffrent sql tables then we should be good. But if we run in to temp space issues on oracle server . Then i have to load the two tables separately which consumes the same time as earlier one.
I was thinking if there is a way in SSIS where I can pull data from same oracle table in to two diff sql tables at same time?
I have multiple xml data file in a directory say C:XMLData abc1.xml, abc2.xml, abc3.xml etc.
Need to loop through each file in ssis with Foreach loop container, and get the file name say abc1, and load the data of abc1.xml to abc1 table in sql server DB.
Next iteration will pick up the abc2.xml and find the abc2 table in sql server DB then insert the data in abc2 table.
While each iteration, xml source should also point each xsd file correspondingly.
 Tables are already created in DB
I solved my problem up to getting the file name from ech iteration and assigned file name to variable, in oledb destination data access mode I select Table or view name variable, then corresponding table will get selected for data insertation.
Just wanted to know how can I read each xsd file for each xml data files while iteration.Â
I've been working on an SSIS package trying to load some data and the archive sequence is faulty. I've been trying to load a few tables created in a previous sequence into a local archive file and I've been getting the error "Could not find a part of the path."
The results aren't telling me what it's finding last and so I don't know where to start.
And the source DOES have data in it. It's something between the source and the destination.
I'm new to my company, although not new to SQL 2005 and I found something interesting. I don't have an ERD yet, and so I was asking a co-worker what table some data was in, they told me a table that is NOT in SQL Server 2005's list of tables, views or synonyms.
I thought that was strange, and so I searched over and over again and still I couldn't find it. Then I did a select statement the table that Access thinks exists and SQL Server does not show and to my shock, the select statement pulled in data!
So how did this happen? How can I find the object in SSMS folder listing of tables/views or whatever and what am I overlooking?
Would using the Restore cmd be the best option to take a backup from one database(db1) and apply it to another database(Edb2) even though they have different names?
I am trying to load many XML files into SQL Server 2005 Standard Edition. Each XML file contains many individual XML items, each starting with: <?xml version="1.0" encoding="UTF-8"?> The files do not have a .xml extension. I do have a schema for the data. I can also get hold of a dtd file for the data.
Since I am new both to XML and to databases I have been trying to follow examples wherever I can find them. I have not found the right example or tutorial. The closest that I found was in Kirk Haselden's book (a very nice book to learn from, by the way). On page 73 he sets up an OLE DB Source. His example works fine. But when I try to use XML Source instead of OLE DB as the source, I cannot get it to work. A typical error that I get reads as follows:
"Error at Data Flow Task [XML Source [1]]: There was an error setting up the mapping. The 'http://www.w3.org/XML/1998/namespace:lang' attribute is not declared."
Evidently I am doing something wrong, but what? Can someone point me to an example of using SSIS to load many XML documents that I could use as a model?
Hi, I am loading Data from Mainframe to SqlServer on WinNt. Normally it was taking 35 mts do the dts job. on last two days it runs for more then six hours still the job doesnt get over. I am at a loss to know what to do and how to fix this problem. THe main frame ppl said sqlServer is fetching the data very slowly. If anyone knows the solution pl post a solution
Hi! maybe this question is very common, but i can resolve that. I have a DB marked 'loading', and nothing seems to change this state. Is any way to revert to a normal state, or the only way is destroy and re-create? Thanks in advance
What is the best way to load large amounts of data? I am working on a project where I will need to load data into approx. 20 tables. Into several of the tables I will need to load around 400,000 records. I am familiar with the concepts involved in using BCP but was hoping I could avoid the step of going to text files. I am pulling data from Access (either 97 or 2000). Any suggestions would be welcome.
I gave the command to restore database and in the middle of recovery i canceled it. now the database is showing 'loading'. I am not able to do any thing to it.please suggest how to make it operational.I also want the data upto the time, I gave the command 'recover database'. thanks.
How would you fix these two problems. 1. On the window of Enterprise Manger and under the databases, the database shows one of the databases like this manag_dat(recovering)rather than manag_dat. (manag_dat is the name of the database). What could cause the probem and how to fix it? 2. Same position as above, but instead it shows manag_dat (loading). I tried to delete its device and the database. it hang. How would it happen and how to fix it?
I've a server X with 100 logins. Peridiocally this X server gets new logins created and gets the users password reset. Aside from this, I've a Y server. what I would like to do is, Have all my logins from X server get created on Y server with the same login/password every end of the day.
I've tried BCP. But i dont know, If BCP will work or not. Is there a way to do this?
I have just restored one of my databases. Now, that database is no longer available to me. It has been marked as 'Database(Loading)'
I have reviewed and tried the suggestions given from previous posts of this same question, and they have not been successful (i.e. 'RESTORE DATABASE DPeople WITH RECOVERY' as well as re-restoring the database).
