Creating Flat File And SQL Server 2005 Hangs...
Feb 10, 2006i m using the hands-on labs for SQL server 2005.
Step 6 age 14 i went thro and it make my SQL server 2005 hangs when i try to use the flat file destination.
i m using the hands-on labs for SQL server 2005.
Step 6 age 14 i went thro and it make my SQL server 2005 hangs when i try to use the flat file destination.
Hi there,
after we've upgraded to SP2 my flat-file-source component hangs in the validation phase.
normally at 50% or 66%.
i see no activity what-so-ever cpu-wise/memory-wise.
have no idea how to fix it.
(sorry if i am posting this twice, couldn't see my previous post)
I'm looking for a manner to create by code a flat file connection manager and a flat file destination.Greets
View 3 Replies View RelatedI have a table that holds in each record an image (varbinary(max) actually), a text reference for the image and a MIME type for the image. I need to read this table and for each record that has been created since the last run, I need to create a file with the image as the content, the mime type as the file extension and the text reference as the file name. There will be one file created per record found by the data flow source.
I was assuming that I could use the flat file destination and manipulate the file naming using the contents of each record in the flow but am completely stumped on how to achieve this.
Does anyone have any ideas?
thanks
Trying to figure out the best method of reading in a number of flat files, all with different number of columns and data types and outputting them to a database.
Here's the problem: They are EBCDIC encoded and some of the columns are packed decimal. I've set up one package that takes the flat file, unpacks the decimal (Using UnpackDecimal component) and then sending the rest through a second component to go from EBCDIC -> ASCII.
What I need is a way to do this for every flat file based on the schema for that flat file. One current solution is to write a script/app to create the .dtsx XML file and then execute that for each flat file. It appears like this may be possible, but I haven't gotten far enough to know for sure. So my questions are this:
1) Is there an easier way to do this (ie somehow feed the schema to the package and use it to dynamically set up the column makers and determine which columns get fed to the unpack decimal component.
2) If there isn't a better way, will dynamically creating the .dtsx XML file based on the necessary input/output columns for each flat file work? If so, what is a good source of information on this (information about how the .dtsx XML file is set up, what needs to be changed/what doesn't, etc).
Thanks,
Travis
I created a SSIS to export to a flat file (from a SQL command : a stored proc).
I don't wan't my SSIS to create an empty file if there is no data.
How can I achieve this ?
Thanks,
Vince
I am new to SQL Server, and migrating part of an Access application toSSE. I am trying to insert a comma delimited file into SSE 2005. I amable to run a BULK INSERT statement on a simple file, specifying thefield (,) and row () terminators. I can also do the same with aformat file.Here is the problem. My csv file has 185 columns, with a mixture ofdatatypes. Sometimes, a text field will contain the field delimiter aspart of the string. In this case (and only in this case) there will bedouble quotes around the string to indicate that the comma is part ofthe field, and not a delimiter.Is there any way to indicate that there is a text delimiter that isonly present some of the time?If not, any suggestions on getting the data into SSE?Many thanks for your input.Cheryl
View 9 Replies View RelatedI'm using SSIS to import seven flat files (each containing a different record type) into a staging database. This part was easy.
Now I need to export the records from all seven tables into a single flat file structured in a nested hierarchy using common keys. (This format is required by the vendor for loading data into a new system).
I could use some ideas on the data transformations needed to combine all seven record types into an hierarchical record set which can then be written to my Flat File Destination. I'm currently looking at an article on SLQIS.com ("Handling Different Row Types In The Same File") which seems close to what I need, but they are importing (ref: www.sqlis.com/54.aspx ). I'm not sure if I should just reverse this for export or use something different. Any comments are appreciated.
Diagram of Record Hierarchy
typeA (parent key, ...)
typeB1 (parent key, childSet key, date, ...)
typeB2 (parent key, childSet key, ...)
typeC (parent key, childSet key, ...)
typeD (parent key, childSet key, ...)
typeE1 (parent key, childSet key, date, ...)
typeE2 (parent key, childSet key, ...)
The record types B1 through E2 form a complete set. Each set has it's own unique child-set key. There may be one or more sets for each typeA record (although it's possible that typeE records don't exist in the most recent set).
Hi everyone,
There is a small problem encountered while creating a package in sql
server 2005.
Actually i am using a flat file which has 820 rows and 2 columns which
are seperated by line feed(for ROW) and tab(for COLUMN).after
importing i found that ther are only 800 rows imported into the table.
Ather verifying the input file i found out that there are some null
values in the second column so there is no line feed for those
values.
Can anyone please help me how to give multiple delimiters for the same
input flat file.
Hi friends, Is there any way to create a text file or word file through sql server 2005.If you have any source please suggest me.. P.Kumaran
View 3 Replies View RelatedHi,
I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.
Did someone else faced this issue?
Thanks,
AQ
Hi,
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
I am transferring data from an OLEDB source to a Flat File Destination and I want the column width for all of the output columns to 30 (max width amongst the columns selected), but that is not refected in the Fixed Width Flat File that got created. The outputcolumnwidth seems to be the same as the inputcolumnwidth. Is there any other setting that I am possibly missing or is this a possible defect?
Any inputs will be appreciated.
M.Shah
Hi all,
I m using SSIS and i am transfering the data from Flat File Source to the OLE DB destination File. The source file contain some corrupt data which i am transfering to the other Flat file destination file.
Debugging is succesful but i am not getting any error output in the Flat file destination file.
i had done exactly which is written in the msdn tutorial of SSIS.
Plz tell me why i am not getting the error output in the destination flat file?
thanx
I have a Rounding error: Between flat file connection manager Source & OLE DB Connection Destination (SQL Server 2005) in my Dataflow.
File looks like this lets call column names Col A,B,C,D
70410000 RD1 1223631.92 196042.42
70329000 ICD 11025.84 3353.88
71167300 COL 104270.59 24676.96
flat file connection manager settings: first row Column names then Advanced tab Col A float , Col B float , Col C string ,Col D float ,
OLE DB Connection Destination (SQL Server 2005)
CREATE TABLE [dbo].[PT_CUST_ABR](
[PARTY_NO] [float] NULL,
[PARTY_NAME] [varchar](75) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
[TELECOMABR] [float] NULL,
[GENIABR] [float] NULL,
Problem: ColA (Source) Rounding error to PARTY_NO (Destination)
I have a field of text of in a flat file that the flat file connection manager Source picks up correctly €ś70000893€?
However when it gets the OLE DB Connection Destination the data has changed to 70000896. That€™s before its even Written to the database.
The only clue that something is wrong in the middle is the great Data viewer shows the number as 7.000009E+07
Other clues looking at the data it appears there is a rounding error on only the number that dont end in 00
ColA (Source) PARTY_NO (Destination)
71167300 71167296
70329000 70329000
70410000 70410000
Any ideas people?
Thanks in advance
Dave
First, a couple of important bits of information. Until last week, I had never touched SISS, and therefore, I know very little about it. I just never had the need to use it...until now. I was able to convert my first 3 flat files to SQL2005 tables by right clicking on "SISS Package" and choosing "SISS Import and Export Wizard". That is the extent of my knowledge! So please, please, please be patient with me and be as descriptive as possible.
I thought I could attach some sample files to this post, but it doesn't look like I can. I'll just paste the information below in two separate code boxes. The first code box is the flat file specifications and the second one is a sample single line flat file similar to what I'm dealing with (the real flat file is over 2 gigs).
My questions are below the sample files.
Code Snippet
Record Length 400
Positions Length FieldName
Record Type 01
1,2 L=2 Record Type (Always "01")
3,12 L=10 Site Name
13,19 L=7 Account Number
20,29 L=10 Sub Account
30,35 L=6 Balance
36,37 L=1 Active
37,41 L=5 Filler
Record Type 02
1,2 L=2 Record Type (Always "02")
3,4 L=2 State
5,30 L=26 Address
31,41 L=11 Filler
Record Type 03
1,2 L=2 Record Type (Always "03")
3,6 L=4 Coder
7,20 L=14 Locator ID
21,22 L=2 Age
23,41 L=19 Filler
Record Type 04
1,2 L=2 Record Type (Always "04")
3,9 L=7 Process
10,19 L=10 Client
20,26 L=6 DOB
26,41 L=16 Filler
Record Type 05
1,2 L=2 Record Type (Always "05")
3,16 L=14 Guarantor
17,22 L=6 Guar Account
23,23 L=1 Active Guar
**There can be multiple 05 records, one for each Guarantor on the account**
and the single line flat file...
Code Snippet
01Site1 12345 0000098765 Y 02NY1155 12th Street 03ELL 0522071678 29 04TestingSmith,Paul071678 05Smith, Jane 445978N 05Smith, Julie 445989N 05Smith, Jenny 445915N 01Site2 12346 0000098766 N 02MN615 Woodland Ct 04InfoJones,Chris 012001 01Site3 12347 0000098767 Y 02IN89 Jade Street 03OWB 6429051282 25 04Screen New,Katie 879500
As you can see, each entry could have any number of records and multiples of some of the record types, with one exception, every entry must have a "01" record and can only have one "01" record. Oh, and each record has a length of 400.
I need to get this information into a SQL 2005 database so I can create a front end for accessing the data. Originally, I wanted one line for each account and have null values listed for entries that don't have a specific record. Now that I've looked at the data again, that doesn't look like a good idea. I think a better way to do it would be to create 5 different tables, one for each record type. However, records 2 through 5 don't have anything I can make a primary key. So here are my questions...
Is it possible to make 5 tables from this one file, one table for each of the record types?
If so, can I copy the Account number in record 01, position 13-19 in each of the subsequent record types (that way I could link the tables as needed)?
Can this be done using the SISS Import and Export Wizard to create the package? If not, I'm going to need some very basic step by step instructions on how to create the package.
Is SISS the best way to do this conversion or is there another program that would be better to use?
I know this is a huge question and I appreciate the help of anyone who boldly decides to help me! Thank you in advance and I welcome anyone's suggestions!
Hi-I have a sql 2005 server (New to me) that I need to get a report out of of type .inputIt needs to be in the format: HeaderRowreportTitile + rowcount()+ DateFixedWidthcol1 (spaces pad this field to 25) FixedWidthCol2FixedWidthcol3..... FixedWidthcol1 (spaces pad this field to 25) FixedWidthCol2FixedWidthcol3..... The columns in the flatfile HAVETO run into eachother, with NO delimiters.If i try and export it as a fixedWidth file, the columns DON'T line up. If I try as a delimited file, I can't do it w/o having a delimiter character in between.I'm creating an SSIS file (New to me) to do this, is this my best shot of getting a flat text file that I can then ftp to another server? thanksDan
View 1 Replies View RelatedI'm a little stuck here. How can I import a flat tab delimited file into a SQL 2005 Express table? The standard DTS and import features are not there :-(Thanks,Casey
View 1 Replies View RelatedProbably a stupid and regularly asked question but I can't seem to find an answer, so here goes,
we have 16 .txt files, some with over 350 columns.
That info from each individual file needs importing to multiple sql tables.
need to look at sql table1 does record exist? if not create new then add in data once its been transformed eg datetime from yyyymmdd into datetime values [managed to get this using derived column] for first 20 columns, otherwise do update for the 20 columns...
then look at sql table2 and repeat for next n columns....
So I was wondering is it going to be better to write this as a dtsx package? if so can you point me to an example
or should I just write the code as part of a code behind page that scrapes the info and does a standard update/insert procedure?
Any help would be welcome.
thanks
Hello everyone,
I have exported data from few tables from my old sql server 7.0 database. Now, I need to import those data into new database which is on SQL Server 2005 Express. How do I do that in 2005 express server? Any idea?
Or, is there any better way to import data for selected tables into new database?
Thanks,
Ujjaval
Hello,
I am having a problem with SQL Server (Express) hanging during simple debugging of CLR code. I created a test project with a user defined function, and call this from the test script. The output log shows that the sqlservr.exe was auto-attached successfully, and then the everything hangs. I have to terminate debugging in order to regain control, and this takes perhaps a minute or two.
Any ideas?
Just purchased a new server, and am trying to install SQL Server 2005 standard, 64 bit. I'm running Windows Server 2005 r2 64 bit.
Setup support files installs fine.
System configuration passes all 14 checks.
Then it moves to install and just hangs there. Eventually, perhaps maybe after 20-30 minutes does it create a message saying that the Install has failed to respond. Retry or cancel.
Hi
I got an access 2002 application front end with a sql server 2005 express back end. Some of my clients are having some difficulties. After using the application for a while, some of the users are finding that the system just hangs up. It usually happens after the front end application has been running for about an hour (sometimes sooner and sometimes later). There are perhaps 1 to 5 concurrent users and I have checked to see if there are any firewalls stalling it (I think I check all of them)- Is there any way that SQL Server 2005 express could be caused to just stall- This even occurs with the odd laptop. All the appropriate protocols are enabled as well. These databses are not very large.
ANY HELP WOULD BE GREATLY APPRETIATED!!!
Thanks
Frank Srebot
I am experiencing extremely slow behavior (no activity for > 30 mins) during a SS 2005 Dev Edition install on XP SP2 on various steps during the SQL Server Database Services step - e.g. during "Setting File Security". This is very annoying.
I've found many postings on this issue indicating that OS calls in this part of the install (e.g. LookupAccountName()) are network speed dependent.
Does Microsoft have any plans to rework this part of the install to rid the need of these calls? I really wish the install would be possible without these calls - it's driving me crazy monitoring this for hours. We've got to roll SS2005 out to a couple hundred developers eventually and this isn't going to cut it.
Thanks for your time on this matter.
Best Regards,
Mike
I am sorry, I am posting this message again, since I did not get anyreply.I want to export a table into a "fixed width" file using SQL 2005import export wizard.This is the version I have:SQL Server 2005 - 9.00.2047.00For some reason it joins all the rows together. For EX: if the tableis like this:Create table Mytable (col1 varchar(50) null, col2 varchar(60) null,col3 varchar (100) Null)Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")It is not exporting every row in a single line. Actually if I open itin "Ultra Edit", it is all in one line.I used to do this regularly with SQL 2000 import export wizard and itexported every row in one line.I looked at the setting:The header row delimiter has {CR}{LF}Code page has 1252 Ansi-Latin.In the Advanced tab:String:dt_str.I tried changing the header row delimiter to just {CR} or just {LF}.Also I tried changing the string to dt_text and nothing seems to help.Please help.Thank you
View 1 Replies View RelatedI just bought a new laptop with Vista pre-installed and was able to install Visual Studio 2005 Pro, but when attempting to install SQL Server 2005 it just hangs at the IIS detection step. I was using the administrative account. I was able to install the Developer Edition of SQL Server 2000, but I need 2005.
Some posts mention IIS version issues and 32 vs 64 bit version issues. Can anyone ellaborate on what is happening and why? What is the work around?
After the installer has asked for authentication mode and has started verifying the authentication it simply hangs. I've let it sit there for an hour before taking the installer forcibly down but nothing seems to be happening except for two osql.exe processes showing up in the task manager.
I've tried both windows authentication and SQL server authentication but the outcome is always the same. At the same time I'm, however, able to login to the database engine and all services with management studio using both authentication modes.
The last few lines in HotFix.log are as follows:
05/09/2006 09:49:09.999 Authenticating user using Windows Authentication
05/09/2006 09:49:10.015 SQL Service MSSQLServer was previously running, ready for authentication
05/09/2006 09:49:10.218 SQL Agent Service SQLSERVERAGENT was previously running
05/09/2006 09:49:12.093 User authentication was successful
This would indicate that there's nothing wrong with the authentication but with whatever is supposed to take place after that.
There are quite a few lines in the log similar to the one below:
05/09/2006 09:47:08.438 Failed to read associated hotfix build information for the following file: C:Program FilesMicrosoft SQL ServerMSSQL.2MSSQLBinnSQLServr.exe
Redist9_Hotfix_KB913090.log seems to be the only individual log file that ends with line saying "Hotfix package launched" whereas all other logs have several lines of checks being completed successfully.
Any ideas what might be causing the installer to hang and how to fix it?
I have two tables
t_DTM_DATA_STAGING around 2 million records
t_DTM_DATA around 251 million records
The below SQL statement looks for records in the t_DTM_DATA_STAGING table that are not in the t_DTM_DATA table and adds them to a 3rd table. (t_DTM_DATA_STAGING2)
This statement has been running fine for weeks, but now it seems to get hung every day. I ran sp_Who2 and it says the status is runnable. I let it run for around 5 or 6 hours the other day to see if it will finish but it didn't. This SQL job is step 3 in a 6 step SQLAgent job that usually finishes in 30 to 45 minutes.
I'm not sure how to troubleshoot this problem. No other jobs are running at the time this job runs.
Could this SQL statement be written a better way?
Thanks for any help anyone can provide.
Jerid
SET QUOTED_IDENTIFIER ON
INSERT INTO
[DTM].[dbo].[t_DTM_DATA_STAGING2]
([CP],,[MAJ],[MINR],[LOCN],[DPT],[YEAR],[PD],[WK],[TRDT],[SYSTEM],[AMOUNT],[DESCRIPTION],[GROUP],[VENDOR]
,[INVOICE],[IDAT],[PO_NUMBER],[DDAT],[RCV#],[RDAT],[RSP],[EXPLANATION],[UPLOAD_DATE],[UPLOAD_USER],[UPLOAD_NAME]
,[RELEASE_DATE],[RELEASE_USER],[RELEASE_NAME],[TRTM])
SELECT
t_DTM_DATA_STAGING.CP, t_DTM_DATA_STAGING.CO, t_DTM_DATA_STAGING.MAJ, t_DTM_DATA_STAGING.MINR, t_DTM_DATA_STAGING.LOCN, t_DTM_DATA_STAGING.DPT,
t_DTM_DATA_STAGING.YEAR, t_DTM_DATA_STAGING.PD, t_DTM_DATA_STAGING.WK, t_DTM_DATA_STAGING.TRDT, t_DTM_DATA_STAGING.SYSTEM, t_DTM_DATA_STAGING.AMOUNT,
t_DTM_DATA_STAGING.DESCRIPTION, t_DTM_DATA_STAGING.[GROUP], t_DTM_DATA_STAGING.VENDOR, t_DTM_DATA_STAGING.INVOICE, t_DTM_DATA_STAGING.IDAT,
t_DTM_DATA_STAGING.PO_NUMBER, t_DTM_DATA_STAGING.DDAT, t_DTM_DATA_STAGING.RCV#, t_DTM_DATA_STAGING.RDAT, t_DTM_DATA_STAGING.RSP,
t_DTM_DATA_STAGING.EXPLANATION, t_DTM_DATA_STAGING.UPLOAD_DATE, t_DTM_DATA_STAGING.UPLOAD_USER, t_DTM_DATA_STAGING.UPLOAD_NAME,
t_DTM_DATA_STAGING.RELEASE_DATE, t_DTM_DATA_STAGING.RELEASE_USER, t_DTM_DATA_STAGING.RELEASE_NAME, t_DTM_DATA_STAGING.TRTM
FROM
t_DTM_DATA_STAGING
LEFT OUTER JOIN
t_DTM_DATA AS t_DTM_DATA_1
ON
t_DTM_DATA_STAGING.TRTM = t_DTM_DATA_1.TRTM
AND
t_DTM_DATA_STAGING.TRDT = t_DTM_DATA_1.TRDT
AND
t_DTM_DATA_STAGING.PD = t_DTM_DATA_1.PD
AND
t_DTM_DATA_STAGING.YEAR = t_DTM_DATA_1.YEAR
AND
t_DTM_DATA_STAGING.DPT = t_DTM_DATA_1.DPT
AND
t_DTM_DATA_STAGING.LOCN = t_DTM_DATA_1.LOCN
AND
t_DTM_DATA_STAGING.MINR = t_DTM_DATA_1.MINR
AND
t_DTM_DATA_STAGING.MAJ = t_DTM_DATA_1.MAJ
AND
t_DTM_DATA_STAGING.CO = t_DTM_DATA_1.CO
AND
t_DTM_DATA_STAGING.CP = t_DTM_DATA_1.CP
WHERE
(t_DTM_DATA_1.CP IS NULL)
I can't use DTS nor DTSwizard as I need to put it in a .sql and run it through a command line via .bat file (it's more for the users).
Each row ends with an EOL character, the fields are all fixed width, but I have a little problem here, some rows are empty but just with a EOL character.
How shall I go about it?
many thanks! :D
HELP!
For the past several weeks, I have been trying to install SQL Server 2005 on a Win XP SP2 PC. Regardless of the options I choose (SQL Engine + Client Tools, Client Tools only, etc.) when the installation gets to the end of the Client Tools setup and the setup status displays "Removing Backup Files", my PC appears to hang.
For a period of time, there is a fair amount of HD activity which gives me the impression that files are in fact being deleted, but then there is no HD activity and the installation appears to hang. Finally I give up and kill the setup process and reboot.
After reboot, it appears that the client tools have been installed (the icons appear and I can open the Management Studio, but when I try to connect to an existing instance of SQL, that hangs. This occurs for either Windows or SQL authentication.
Has anyone experienced this behavior and determined the root cause? What EXACTLY is happening during the "Removing Backup Files" phase of the installation?
Thanks for any help in advance,
Marc Mueller
I have a virtual machine that previously had SQL Server 2005 installed. Since I changed the computer name, I have to reinstall SQL. I removed all the SQL components and am trying to reinstall the database and reporting components. The installation is failing on the "Removing temporary files" step. According to the setup progress screen, the following components have been installed:
OWC11
SQL Server Bakcward-Compatibility Files
SQL Server Database Services
Reporting Services
Visual Studio Integrated Development
SQL Server Books Online
SQLXML4
Workstation Components, Books Online - this step is still in "configuring components" status.
The virtual machine is running Windows 2003 Server SP1 with all current patches, using just under 2 GB of memory (1920 MB of RAM), the host is Windows XP SP2 with all current patches, with 4 GB of memory.
I had no problem doing this procedure on a different virtual machine, so I'm not sure what the problem is here.
How can I take this example Flat file and parse out each section to a new flat file? Each section starts with HD (header row)
http://www.webfound.net/flat_file_example.txt
e.g. an example output file based on above (cutting out the first section) would be:
http://www.webfound.net/flatfile_output.txt
Also, I'll need to grab a certain value in each header row (certain position in the 100 byte header row) to use that as part of the filename that's outputed. I assume it would be better to insert these rows into a temp table then somehow do a search on a specific position in the row...but that's impossible? The other route is to insert each row into a temp table separated out by fields but that is going to be too combursome because we have several formats to determine separation of fields based on the row type so I'd have to create many temp tables and many components in SSIS when all we want to do is again:
1) output each group (broken by each header row) into it's own txt file
2) use a field in the header row as part of the name of the output txt file (e.g. look at the first row, whcih is a header row in flat_file_example. txt. I want to grab the text 'AR10' and use that as part of the filename that I create
Any suggestions on how to approach this whole process in SSIS...the simplest approach that will work ?
For a database, we have 4 data files in a particular file group and the file sizes are almost 70 GB each.
Do I come across any performance issues if I create/pre-allocate an additional data file in the same file group so that the existing files don't grow too much?