DTS - Import Flat Textfile Into Two Separate Tables
Jul 20, 2005
I have a report that's created each day as a flat textfile.
Because I came from the Access world, I created a macro that imports
it with a schema that gives meaningful names to the various columns,
and then uses a query to massage some of the data for me (deletes the
first blank row and does a couple of calculations)
Then I use DTS to import the Access query as a table.
the textfile has a column called "File_num", and among several others,
a column called "Serial_num". (the file numbers represent shipments,
and sometimes there are more than one serial number in the shipment,
etc., so there is a separate line for every serial number)
Naturally, I would like to split this info into two tables..one that
does not contain the serial numbers and has a primary key on the
"File_num" column, and another table that would contain just the
"File_num" and "Serial_num" columns. That way I could relate them
later...but most importantly, it will give me a table where I can use
the "File_num" as my primary key.
What would be the best way to import these two tables from one source
textfile? The other thing that gives me problems is that the text
file has no column names, and the first row is always blank.
I'm very new to SQL and DTS and would appreciate any direction.
We are trying to use SSIS Import export wizrd to import the flat files (CSV format) that we have into MS SQL Server 2005 database tables. We have huge number of CSV files. Is there a way by which we can import these flat (CSV) files in to corresponding SQL server tables in a single shot. I would really appreciate this help as it is painful to convert each and every file using the Import Export wizard.
Can somebody please help me with the following problem.
I want to import data from a textfile called "Links.txt" into a SQL-server database called "LinkData". The data in this textfile is separated by pipelines. And this import should be done every 2 hours automatically. How can I make a proces or something in SQL server 7 that will fill the database with the data out of this textfile every 2 hours. Please help me. I really don't know.
I want to import data from a textfile called "Links.txt" into a SQL-server database called "LinkData". The data in this textfile is separated by pipelines. And this import should be done every 2 hours automatically. How can I make a proces or something in SQL server 7 that will fill the database with the data out of this textfile every 2 hours. Please help me. I really don't know.
I was planning on using DTS for this Only I really don't know how. I tried it with "Data driven query task" in combination with "Text file (source)" and "Microsoft OLE DB Provider for SQL Server". But when I run it, it gives an error. What am i doing wrong??? Help me please
I am trying to add 2 separate columns from separate tables i.e column1 should be added to column 2 when inserted and I want to use a trigger but i don't know the syntax to use...
I have a flat text file with lots HTML tags and corresponding valuesÂ
For example <Near_Side> 5563 </Near_Side> <Top_Down_Code> Xe345 <Top_Down_Code>Â
So, For example I can use the a functionÂ
ALTER function dbo.StripHTML( @text varchar(max) ) returns varchar(max) as begin   declare @textXML xml   declare @result varchar(max)   set @textXML = REPLACE( @text, '&', '' );
[Code] ...
To which Select dbo.StripHTML('<Near_Side> 5563 </Near_Side>')  ValueÂ
I'll get 5563
However how would you get the values within the tag itself ? E.g.Â
Value Name 5563 Near_side Xe345    Top_Down_Code
I was thinking along the Charindex but cant seem to get it right.
And i want to use SSIS package dynamically load data from database into three separate flat file, each table into each file.
I know i got to use for each loop task ADO.Net schema row set enumerator, with OLEDB connection manager, select table name or view name variable from access mode list, but the problem comes, as table name is dynamic then flat file connection is also dynamic, i am using visual studio 2013...
I am new to T-SQL and triggers Any help will be appreciated
I am trying to change this code to insert firstname, surname (taken from employee table on db A) to firstname, surname on customer table of DB B but also create cust_id on customer table and DB B. currently I am getting all rows of customer.cust_id filled with the same data whenever a new data is inserted into (firstname,surmname of employee table)
Create trigger gen_cust_id ON employee for insert AS Update customer SET cust_id =( SELECT Replicate('0',(4-DATALENGTH(CONVERT(varchar(10),i.id)))) + Convert (varchar(10),i.id) + Substring(i.lastname,1,3) + Substring(i.firstname,1,1) from employee C INNER JOIN inserted i on i.id=c.id) from employee C INNER JOIN inserted i on i.id=c.id
How would i write a single sql statement where i can get that counts how many bookIDs are listed for each custoemrID and how many magzaineIDs are listed for each customerID and have it return one table that looks like this:
We have some tables that we have spread across two databases. The segregation isn’t essential, but the entities involved were disparate enough that we thought it made sense. However, our client app regularly & frequently requires information that can only be answered by queries to tables in both databases. It has been suggested that segregating the tables as we have introduces a performance hit. At this stage, it would be relatively easy to re-combine the tables into one DB.
Hello, I am working with a database that among other things uses multipart keys as the unique indexes which are not consistent from say one table where a parent record resides to another table which contains related child records. For example I am working with two tables right now, one that contains content that I'll call Contents and the other which contains Usage information about the contents (number of view, a rating and comments give by a customer) which I'll call ContentsUsage. The system that manages the data for the tables has a versioning system by which, whn a content item is added (first time) a "unique" id (guid) and a version number of 1 is created along with the rest of data items in the Contents table and likewise in the ContentsUsage table (essentially a one to one mapping) on the like named fields in that table. Now, each time a given record in the Contents table is updated a new version, with the same guid is created in the Contents and ContentsUsage table. So one side I have:ContentGUID > AAAAVersion > 1ContentGUID > AAAAVersion > 2And the other table (ContentsUsage)ContentGUID > AAAAVersion > 1ContentGUID > AAAAVersion > 2 While both of these tables have a quasi-unique record (row_id) of type char and stored as a guid neither obviously are the same in the two tables and having reviewed the database columns for these tables I find that the official unique key's for these tables are different (table 1, Contents combines the ContentGUID and Version) as the composite / mutli-key index, while table ContentsUsage uses the RowGUID as it's unique index. Contents RowGUID (unique key)ContentGUIDVersionViewsRatingComments................RowGUID ContentGUID (unique key)Version (unique key)Description..... Bearing this in mind I am unable of course to link directly the two tables by using the just the ContentGUID and have to combine the additional Version to I believe obtain the actual "unique" record in question. The question is in terms of writing queries, what would the most efficient query be? What would be the best way to join the two in a query? And are there any pitfalls with the current design that you can see with the way this database (or specifically these tables are defined)? It's something I inherited, so fire away at will on the critique. Having my druthers I would have designed these tables using a unique key of type int that was autogenerated by the database. Any advice, thoughts or comments would be helpful. Thanks,P.
I've read that if particular tables are frequently queried together through a join then these tables should be placed on different devices on different physical disks. What does this mean exactly and how would you configure this? Is this a common practice in high-performance real-world environments (or should it be)?
I would like to 'one table' record to separate 'two or three tables' . I just know use the DTS , try to import and export again and agian. So trouble.
Could you give me some suggestions for me? For example , 'Cursor' write in new table . But I try to SQL Server Books Online which is not suitable for me solving problems. One table separate two or three tables. Can you wirte the detail example for me? Thx a lot.
Hi, We are building an application for online system for people to place ADs for selling various used items like Car, Electronics, Houses, Books etc. If someone selling a car then he can fill out headline, year, make, model, mileage, transmission, condition, color, price, description, contact etc. Similarly if someone selling a digital camera he will fillout headline, memory, zoom, megapixel, maker, model, color, batter, description etc. Option 1: I can have a main table to hold the common attributes of all different types of ADs (headline, images, contact, price, color, condition, description) + 1 table to store string values of all ADs (car: maker, model, square feet (if house), memory, megapixel (camera) etc) + 1 table to store the droplist select values(car: transmission, door, seat etc; house: year_built) pros: single table for all ADs. unique IDs for all ADs, easy to extend as new attributes can be dropped easily. cons: lot of physical reads of 2nd and 3rd table from join. 10 times physical reads compared to option 2 when reading 5000 records. Option 2: have different set of table for each AD type. Car will have its own main table + 1 table to store multiselect list box values. Similarly housing will have its own set of tables pros: 10% less physical read than option 1. cons: hard to add new attributes. We have to modify the main table by adding one column. Query will go to different table based on the category. Do you have any suggestions on which way to go?Thanks
We have a large Datawarehouse and the size is 50TB.. The tables are placed in filegroups based on the schema like fact, dimensions, raw data each sit on seperate filegroups. I am thinking will it make sense to seperate the large facts which are having billions of rows so that they reside on filegroups on their own..
I am trying to tie together tables that show quantities of a product committed to an order and quantities on hand by a location.
My end result should look like the below example.
Item Location QtyOnHandByLocation SumQtyCommitTotal Prod1 NJ 10 10 Prod1 NY 10 0 Prod1 FL 0 0 Prod1 PA 0 0
So I can see I have 10 items in NJ On Hand and Committed to an order. 10 available in NY but not on an order. Then the other two locations have no quantities.
Below is the CTE but it produces inaccurate results. I've tried running it several different ways by playing with the grouping but have no luck thus far.
--create the temp table Create table #SalesLine ( Novarchar (50) not null , LocationCodevarchar (50) not null , QtyCommitint not null ) create table #ItemLedgerEntry
[code]....
I am close to the desired results but can't find a way.
Hello, I'm trying to import data from a flat file into a table that has smalldatetime data types. I tried creating triggers on the smalldatetime data types that converts the data from a string to a datetime value but the import is still unsuccessful. What should I do?
Col002 looks like this in my flat file 'ex: 20000112' DTSDestination("entry_dt") = DTSSource("Col002") I get an error when trying to put the value of col002 into entry_dt.
I am trying to import a flat file with large rows into MS SQL Server. This flat file consists of about 100 columns of data, followed by a set of 10 columns repeated 50 times.
I would like very much to break the data apart in the import. What's the best way to handle it?
Hi, From the filemaker database, I am exporting data to the Flat file separated by tabs. From this Flat file I need to import into Sqlserver table. The Flat file has the 23 columns and 4000 rows. To insert into Sqlserver table I am creating the Flatfile connection. In the dataflow tasks, I am creating the Flatfile source, OLEDB destination and mapping the columns. And to run the package I am doing the Debug->start with out debugging. In my flat file I have columns like SSN, Email. So in Sqlserver table I am defining the columns as the Nvarchar(200). If I insert the 6 rows data from flat file to Sqlserver, this works fine.
If I tried to insert the 23 rows data, by clicking on Debug->Start debugging -> no data inserted into Sqlserver table. the arrow between flatfile source, OLEDB destination is showing the 450 rows. How can I view the errors?
When I mouse over on the OLEDB destination, it is showing "truncation occurs on the column (20th column) more than 512 characters.." Initially the Output parameter length is 50 and datatype is Unicode char, I increased it to 512. What is the problem here? If there are more columns I am not able to insert data. How to view the errors?
I am trying to import a flat file using SQL Server Management Studio and am receiveing the error: Error 0xc0202055: Data Flow Task: The column delimiter for column "Column 19" was not found. (SQL Server Import and Export Wizard)
I would like to capture the rows that are causing the error and have the import continue. Am I able to edit the behavior somehow?Thanks.
In database DB1, I have table DB1.dbo.Suppliers1. This table has an ID column of type INT named SUPPL1_ID
In database DB2, I have table DB2.dbo.Suppliers2. This table has an ID column of type INT named SUPPL2_ID I would like to update DB2.dbo.Suppliers2 based on values from DB1.dbo.Suppliers1 joining on SUPPL1_ID = SUPPL2_ID.
How can I do this in SSIS?
Assumptions:
linked servers are not an option, as I want the SSIS package to be portable and not dependent on server environments. TIA.
Request is to merge or join or case stmt or union or... from up to four unique columns all in separate tables to new combined table (matrix) of results from said.
I am trying to import a flat file into SQL Server 2005 using SSIS. I have never used it before and I am getting confused by the error I am receiving.
I have a link to a flat file, that gets sent through a Derived Column flow where dates in YYYYMMDD are changed to MM/DD/YYYY format. Then the string MM/DD/YYYY is converted to a date in a Data Conversion flow. And finally the data is put into a SQL Server table (currently with no rows).
The problem I am having is with a text field with the email address in it. The error I am getting is:
[Import Allstate Auto Club [1]] Error: Data conversion failed. The data conversion for column "email_source" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
The problem is I can't see where in the flow the problem is. The field length is 20 wherever I look and the codepage is 1252 wherever I look. Does anyone have an insight? Keep in mind, I have never used SSIS before and I consider myself an amateur with SQL Server. It could easily be a data type conflict or something easy. Any help will be appreciated.
I have a fixed width flat file I'm trying to insert into an SQL 2005 table using SSIS -- it's a recurring task. One of the columns in the flat file has to go to a column of type Numeric. No matter what I try : a data conversion, defining the field as DT_NUMERIC in the connection,... I always get "The conversion returned status value 2 and status text :The value could not be converted because of a potential loss of data". It is driving me bonkers, up to the point that I find myself wishing for the 'good old' DTS days of SQL 2000. And I dread to think what will happen when I try to port some serious, much more complex DTS packages on my SQL 2000 to SQL 2005.
The data in question represents longitudes and latitudes so quite often there is a leading white space in the data : ex. : " 95.15". Surely that cannot be the cause ?
I've spent hours doing the RTFM-thing and searching the newsgroups, fora...you name it. Apart from ending up running in circles in the MS documentation, the only thing I've really learned so far is that I'm aparently not the only one driven to dispair by the new SSIS thing.
I can think of a number of ways to hack my way around this thing, but that's not the kind of 'progress' I had in mind when I started the move to SQL 2005.
Boy, do I need HELP! Have a simple csv file that I need to import. Worked fine in sql2000; I put it into dts to execute on a monthly basis. Makes connection, db connection, table creation fine, but stops at validation of flat file?
Basically, I want to go out and get a flat file, drop the existing table, and create the table, and import the information from the flat file. Not a complicated table of about 30,000 records.
Create table [db].[dbo].[tblPatient] ( [patientID] into not null, [chartID] varChar(15) null, [doctorID] int null, [birthdate] datetime null, [sex] varchar(1) null, [raceID] int null, [city] varchar(100) null, [state] varchar(2) null, [zip9] varchar(9) null, [patientTypeID] int null, [patName] varchar(100) null)
Below is the error report that tells me NOTHING!
Operation stopped... - Initializing Data Flow Task (Success) - Initializing Connections (Success) - Setting SQL Command (Success) - Setting Source Connection (Success) - Setting Destination Connection (Success) - Validating (Error) Messages * Error 0xc00470fe: Data Flow Task: The product level is insufficient for component "Source - pmPatientInfo_csv" (1). (SQL Server Import and Export Wizard) * Error 0xc00470fe: Data Flow Task: The product level is insufficient for component "Data Conversion 1" (71). (SQL Server Import and Export Wizard) - Prepare for Execute (Stopped) - Pre-execute (Stopped) - Executing (Success) - Copying to [fhc].[dbo].[tblpatient3] (Stopped) - Post-execute (Stopped) - Cleanup (Stopped)
I'm going to be getting several flat files that need to be imported into one of two tables. Although the text files will have different file names, they will have either "Header" or "Detail" in the file name, so I can tell which table they need to be imported into.
The problem is I don't know enough about SQL Server 2005 to set up an automated import of these files into the database. Does anyone have any suggestions on how to do this, or have any experience in setting this up? Are there any inexpensive programs to load data realtime or on a schedule? (I work for a young company with a very tight budget).
I had an idea of creating a windows vb or batch program to create the import commands for each of the flat files, but I can't find the line command to build the import command.
I have a flat file that uses tabs as the column delimiters and cr-lf as row delimiters. The first portion of the file consists of only two columns for approximately 10 rows and then the file changes to 4 columns for the balance of the file, about 21 rows. The column names are in the first column and the data of interest is in the second column for the first 10 rows and then in the third column for the last 21 rows. Is it possible to set up something like this for parsing in SSIS? I've tried using two columns in the data flow task but then I get columns 1 and 2 through the whole file. If I tell it there are 4 columns in the file, it appends rows to each other so that there is a total of 4 columns in the first 10 rows. This reduces the row count to less than 10 and the data in these rows isn't in the proper place. Is there a way to handle this file in SSIS?
I want to import a flat file to my sql server database. My sql server and web server are on different machines. I used a bulk insert to import the data using a Format.. But now since the sql server and the web server are on different machine it doesnt load the data to the sql server..
i have tried giving http://Ipaddress/Path and that didnt work.. tried mapping the network drive to the webserver and then specified the location and that didnt work too..
I found this connection string on the internet Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c: xtFilesFolder;Extended Properties="text;HDR=Yes;FMT=Fixed";
but i am not sure how to give a ~ delimited and specify a format file..
I€™ll truncate the destination table before each import.
Is the best way to create a temp table; get it populated then update the destination table? Or would the use of Merger (Merge Join) be the better approach.
I have a fixed width flat file that I'm trying to import, and I'm just about there. The last column that I'm struggling with, is a decimal amount. The data in the column looks like this 00000000500 and I need to dump it into a column as 5.000 In otherwords, the data in the file does not have any decimals, and I'm putting it into a sql server column that has the datatype numeric(11,4) I've set the InputColumnWidth to 11, the DataPrecision to 11 and the datascale to 2, and the value is still being imported as 500.000 Is there any way to achieve this other than using a script component to calculate the value? Thanks!
I wanted to know if there was a way to import data from a flat file without specifiying the delimiters. I want to import each line in one row so that i can use the substring function to break of the data as an when i want and not as per the delimited format file or the wizard.
i.e if row one had "abc"|"1453"|"Jack"|"Smith"| etc.... rather than importing these as different columns and rows. I want this all in one row, one column.
