Hi,
is there any way to change the delimiter in a FlatFile Source Adapter dynamically? You can do that with the row delimiter but there is no expression for the column delimiter... Is there any other way besides changing the XML code?
Thanks,
hi all,
is there a way to define a row delimiter to fixed width files? such as this one:
1122333
4455666
7788999
in this file i have 3 columns that are fixed (col1.width = 2, col2.width = 2, col3.width = 3) but have {CR}-{LF} as the row delimiter. when i try to create a flat file connection to these kind of files- he always reads the CR-LF as part of the file text, and there is no place where i can define the row delimiter if i have.
thanks for your help!!!
Sorry if this thread has been repeated but I cannot find a reference to my problem. I have read that apostrophes can be qualified with double quotes but that doesnt solve my problem. I am using SQL 2000 on a Windows 2000 Server trying to set up a job to email the results of a query. I have had no trouble getting it to run as scheduled but cannot seem to get the results out when I add a clause that filters out a text column.
This is the query I am trying to run from a step within a job inside our SQL Agent:
exec master.dbo.xp_sendmail
@recipients = 'user@mycompany.com',
@subject = 'Overnight Safedata Statistics',
@query =' SELECT u.username as BackUpSet , r.requestresultstatus as Status, r.requesttype
from users u ,requests r where u.userid = r.userid
AND requestdate between getdate()-1 and getdate()
AND requestresultstatus <> 'Success' ',
@message = 'Results over the last 24 Hours',
@attach_results = FALSE,
@dbuse = 'BackupServer'
I can successfully run the @query string in query analyzer and it worked fine from SQL mail until I added the last clause to the step (AND requestresultstatus <> 'Success' ). I do not know how to get the 'Success' treated as text.
Any advice would be appreciated.
Thanks
JT
P.S. I am a network guy trying to give our SQL developers a break.
I am getting an error importing a csv file both using SSIS and SSMS. The csv is comma delimited with quotes for text qualifiers. The file gets partially loaded and then gives me an error stating The column delimiter for column "MyColumn" was not found. In SSIS it gives me the data row which is apparently causing the problem but when I look at the file in a text editor at the specific row identified the file has the comma delimiter and it looks fine. I am using SQL Server 2008.
View 9 Replies View RelatedHi,
I have a dts package that imports a number of text files into a SQL Server 2000 database table. The package has been set up to accept a text file with a row delimiter of carriage return and line feed ({CR}{LF}). Some of the text files I receive only have a line feed ({LF}) as the row delimiter and the dts package fails the file. Is there an activex script i can use that will scan the file and change the row delimiter as required?
i was going to use the filesystemobject which allows me to read a line at a time, however the Readline method doesn't read the new line character. The text files are too big to read into one variable and then do a replace.
Any help would be appreciated
I am using SQL Server 2000 SP3, Windows Server 2000 and Windows XP Professional. All systems are fully patched
Regards Justin
I'm in a flat file connection manager editor and I have a flat file where the columns are separated by a space. Does anyone know how to specify a space in the column delimiter option? I've tried {' '} and {s} but these don't work. Not sure what the syntax is for indicating a space. Thanks in advance
View 1 Replies View RelatedHello Everyone,
Hope someone will be able to help me out here.
I have a text file exported from my DTS package and it requires an '!' as a custom column delimiter.
Does anyone have any idea how I can use the '!' mark instead of the Tab or Vertical Bar as my Column Delimiter?
Would appreciate any suggestions.
Thanks,
Kay
hi guys,
i am working on a project witch involves creating packages on-the-fly to import data from txt/csv/xls files according to some definitions on the database.
so far, i have been doing fine.
now we are planning the ASP.net page that enables the customer to define the input file format that will be imported to the system. we want it to have the same listBox as the FlatFileConnectionManager Editor has to define some properties, such as - column delimeiter.
the code to set the column delimiter looks like this:
SSISRunTime.IDTSConnectionManagerFlatFile90 myFilecn = null;
myFilecn = (SSISRunTime.IDTSConnectionManagerFlatFile90)package.Connections["InputFileConnection"].InnerObject;
DtsConvert.ToConnectionManager90(package.Connections["InputFileConnection"]);
SSISRunTime.IDTSConnectionManagerFlatFileColumn90 col
col = myFilecn.Columns.Add(); //.....
string colDelimiter ="|" ; // it actually gets the data from the database... but it is the same thing
col.ColumnDelimiter = colDelimiter;
when we deal with the simple characters- "," , ";" , "|" ... we have no problems with setting the delimiter. but how can i set the delimiter to Tab? or {CR} ? {LF}?
i tried to look at the dtsx- XML , and i see that the column delimiter that is defined when i choose Tab is: _x007C_, but when i try to do something like this:
col.ColumnDelimiter = "_x007C_" ;
it doesn't work. the same happens when I try "{t}" or "Tab".
how do i solve this problem, and enable the user to select Tab as a column delimiter?
Thanks!
Good day everyone,
I have a package that reads data from a CSV file, transforms it and finally loads it in a destination DB table.
My current problem lies in the parsing of the input flat file. I shall illustrate it using a small example.
Source File:
P;Product-1;Short Description for product 1
P;Product-2;Short Description for product 2
Problem:
I configured the flat file connection manager to use semicolon as the column separator. But then I have received some sample flat files where I found that the semicolon might be sometimes used as content of a column data.
Possible Solutions:
I have thought about 3 different solution and I would like to get your feedback and recommendations about them.
Alternative 1:
Use a complex column delimiter, which wouldn't be used in the data.
Example:
P#~#Product-1#~#Short Description for product 1
P#~#Product-2#~#Short Description for product 2
Question 1:
- Is it possible to define such a customized column delimiter for the Flat File Connection Manager?
- If yes, how can I do this?
Alternative 2:
Use double quotes around the data, which the Flat File Source Adapter must somehow recognize and trim before pushing the data down the Data Flow.
Example:
"P";"Product-1";"Short Description for product 1"
"P";"Product-2";"Short Description for product 2"
Question 2:
- Is it possible to configure the Flat File Source Adapter to work as described?
- If yes, how can I do this?
Alternative 3:
Use a Script Component and write the needed code for parsing the Flat File.
Question 3:
- Do you have further suggestions/ideas for solving this parsing problem?
Thanks in advance and my regards,
Samar
I am building a generic SSIS where it takes a text source file and converts to a destination source file . However in the process I want to set the row delimiter and the column delimiter dynamically through the package variable. So that I can use it for any file transfer.
I saw the option for row delimiter in the file connection string property but did not see any column delimiter option.
I am trying to import data from a text file to SQL database through a DTS package. i am always getting "Column Delmimeter not found" error at record number 8000. i copied the 8000th record into the beginning of the file to test if there is any problem with data but still i got the error at 8000 th record. i know it was a problem before and was fixed with sp1. i installed SP4 and still gettting the same error.
View 4 Replies View RelatedI have text data files from a third party and they use comma as field delimiters and enclose the text for each column in double-quotes. Not a problem for most of the data files until they start sending files where there is " within the column values. SSIS package fails with the error:
The column delimiter for column "Column 1" was not found.
Any ideas on how to resolve this issue will be greatly appreciated.Thankspcp
Hi Craig/Kamal,
I got your email address from your web cast. I really enjoyed the web cast and found it to be
very informative.
Our company is planning to use SSIS (VS 2005 / SQL Server 2005). I have a quick question
regarding the product. I have looked for the information on the web, but was not able to find
relevant information.
We are getting Source data from two of our client in the form of Excel Sheet. These Excel sheets
Are generated using reporting services. On examining the excel sheet, I found out that the name
Of the columns contain data itself, so the names are not static such as Jan 2007 Sales, Feb 2007 Sales etc etc.
And even the number of columns are not static. It depends upon the range of date selected by the user.
I wanted to know, if there is a way to import Excel sheet using Integration Services by defining the position
Of column, instead of column name and I am not sure if there is a way for me to import excel with dynamic
Number of columns.
Your help in this respect is highly appreciated!
Thanks,
Hi Anthony, I am glad the Web cast was helpful.
Kamal and I have both moved on to other teams in MSFT and I am a little rusty in that area, though in general dynamic numbers of columns in any format is always tricky. I am just assuming its not feasible for you to try and get the source for SSIS a little closer to home, e.g. rather than using Excel output from Reporting Services, use the same/some form of the query/data source that RS is using.
I suggest you post a question on the SSIS forum on MSDN and you should get some good answers.
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
Thanks
Craig Guyer
SQL Server Reporting Services
Hi gurus,
I've created a linked server (and set up the corresponding schema.ini file) in order to perform bulk-inserts from some CSV text files into SQL tables (from my standpoint the text files are just for reading purposes). The linked server works fine (I can select the data in the files without a problem).
Now the question: is possible to automatically detect when one or more of those files change in order to start the import process automatically? Something like having a trigger created on the CSV files Or there's no easy way to do that so I have, to say something, to create a Job that periodically checks if the files have changed programatically (say, recording each file's timestamp everytime is imported and comparing the recorded value with the current one, or whatever)?
Thanks a lot in advance!
I need to write a genric CSV importer. one set of CSVs define the formats of all teh other CSVs. The format of the second set of CSVs is not know at the time of creating this project/SSIS package.
Imagine having two sets of CSV files.
Each file in the first set of CSVs define the format of files in the second set of CSVs
The first set always have the same format. Each row of the CSV file defines a field with a Name and Type.
The first two columns of each row of in the first set of CSVs have a FormatName and index.
So a simple file may have:
Format1,1,Name,string
Format1,2,Surname,string
Format1,3,DOB,Date
Format1,4,Email,string
The second set of CSVs contains records that have to comply with the format define in the first set.
So in this aprticular case, a CSV file in the second set may have records such as:
Format1,John,Doe,2007-01-01,john@doe.com
Format1,Jane,Doe,2007-02-02,jane@doe.com
The problem I have is creating a generic SSIS package for this.
The first task, loading the first set of CSV file is fairly simple. The CSV format is known.
But the second task is a bit trickier.
Assuming I have SQL tables to load the data.
One 'Fields' record for each row in the CSVs from the fiirst set.
One 'Rows' record for each row in the CSVs from the second set,
One 'Values' record for each value in each row in the CSVs from the second set,
Something like:
TABLE Fields (
FieldID int IDENTITY,
FormatName varchar(100),
Position int,
ColumnName varchar(100),
ColumnType varchar(20) )
TABLE Rows (
RowID int IDENTITY )
TABLE Values (
ValueID IDENTITY,
RowID int,
FieldID int,
ColumnValue varchar(100) )
What would be the best/easiest SSIS approach:
- Dynamically create a SSIS package based on the content of each CSV fiel of the first set and execute that package for each file in the second set, selecting the correct package to execute (all records within a CSV file belong to the same format).
- Write a single SSIS package that iterates through the rows of the second CSVs, does a lookup for each value of each row to find the field name and make an insert into the DB
- Other SSIS method?
- Don't use SSIS to parse the CSV and call a custom C# task?
- Don't use SSIS at all ?
Thanks
I am new to SQL7 and I am trying to run a package that will import fixed text files in a folder and transform the data to a table. I am using a good example found in sywink.com for dynamically importing files in a directory, but I am getting an error message when there is more than one file in the folder. Also, I am able to display all the files during the loop process, but the file initially set for the source connection is the only one that transfers data. I have the close(Transform) connection checkbox option checked, and have tried other methods of closing the connection before the new source name is called, none have helped. Does anyone have any solutions to this type of problem? Or know of other methods used for my situation?
Thanks
James
Hi,
Trying to dynamically set the connectionstring property of the excel source.
This is what I enter instead of the hardcoded excel file paths:
"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + @[User::FileName] + ";Extended Properties=Excel 8.0;HDR=YES"
I get this error, every time I set the delay validation property of the dataflow tab to true.
Cannot detach from one or more processes.
The object invoked has disconnected from its clients
Do you want to terminate them instead?
Thanks
Hi. I am using DBF files as sources for some tables in SQL Server. The problem is that the table names in the DBF files are not all the same (e.g. frame061, frame949). I want to know the names of the tables inside the DBF files. Is there a way to query the table names, something like "select table_name from information_schema.tables" in SQL Server? By the way, those DBF files came from FoxPro. Thanks!
View 4 Replies View Related
Hi All,
I have a challenge i am trying to overcome, hopefully soneone would have come across this issue before..
I am creating a DTS package that will be scheduled to run at a certain time everyday. A source folder exists that get a set of new files everyday.The DTS Package will then read each file and copy the data into a load table in my database the challenge is this:
I am trying to load files from a source folder into my load table, Within each file, the entires are in a specific format using pipes to seperate the data that goes into which column e.g
example of a file entry:
column1 | column2 | column3
data1 | data2 | data3
data1 | data2 | data3
data1 | data2 | data3
And now i am using DTS to specify the file format and map the cloumns as apprporiate to my table...all this is well and good, but my problem is each file has a different name as well as being timestamped, now how do i use DTS to specify the source folder, open each file sequentially and read (or more appropriate, copy the entries into my table, inserting new data from each file into my load table as well as overwriting old data in the load table from the files in the folder ?) is there a way in specifying your source folder in DTS rather than specifying the file in the Menu options (in the transformation data task properties )given, and or do i need to write a script for this(reading the file?)
can someone please give me a solution and how to approach this?
thanks in advance
Hi there,
I am a new member of this site and I am not very much aware of T-sql's working.
My question is what if I need to get one column's data to be the heading of another column.
To be very exact I have a school's database. The table I am talking about is of the results of students. The table contains Student ID, Subject ID, Total marks of the subject, Marks obtained in the subject. Now I want to print a report by generating data from this table. Right now the data is something like this
StuID - - - SubID - - - -Tot - - -Obt
1 - - - - - - -1 - - - - - - -50 - - - 38
1 - - - - - - -2 - - - - - - -50 - - - 41
1 - - - - - - -3 - - - - - - -50 - - - 42
1 - - - - - - -4 - - - - - - -50 - - - 40
2 - - - - - - -1 - - - - - - -50 - - - 35
2 - - - - - - -2 - - - - - - -50 - - - 40
2 - - - - - - -3 - - - - - - -50 - - - 42
2 - - - - - - -4 - - - - - - -50 - - - 41
StudentID and SubjectID fields are related to other tables so I can get the names from there but when I need the report I need the data in the form of
StuID - Sub 1 - - - Sub 2 - - - Sub 3 - - - -Sub4
1 - - - - 38 - - - - - - 41 - - - - - - 42 - - - - - - 40
2 - - - - 35 - - - - - - 40 - - - - - - 42 - - - - - - 41
The Subjects can be different for different students so the query should be dynamic instead of hard coding the names of the subjects. I hope I am clear with my question. The subjectIDs or their names will become the headings and they will contain the obtained marks for that subjects in their columns just for the reports. I have also checked the PIVOT function but was not able to do what I wanted.
Thanks.
Hi Evry one,
I Have Multiple Flat Files in Source Folder(They have Naming Conventions With Todays Date ex: Flatfile_20082204_1,Flatfile_20082204_2,Flatfile_20082204_3 ),
I need to Extract Each and Evry file by Dynamically, and Transform the Flat File then load that Flat file into the Destination Folder with Standard Prefix and Todays Date with a Sequence No ex:Flatfile_20082304_A,Flatfile_20082304_B, Flatfile_20082304_C
Please HELP Me
Thanks In Advance.
I am working on FTP TASK in SSIS Package. i have to get files from FTP that file names are like 20141110.txt. i want to download any particular date file from ftp. How to i set expression in Remote path?
View 3 Replies View RelatedHi,
My scenario:
I have 4 different flat files types each having different no. of column, order of columns etc. I want to upload all the 4 types into the same destination table in the SQL database. Before uploading I need to apply transformation to each column in the flat files. The transformations could be like
1) Multipying the source column by 100
2) Put an if condition for 2 source columns and then select one column to be copied into the destination.
I have the flat files schema with me and also all the transformations that are required.
Question:
Can SSIS provide me with a component that can read the flat file schema and the transformations from the database and apply them to the source data and then upload it to the constant destination table? Can derived column transformation be provided with the input columns list and the transformation to be done on each dynamically?
Why I want this way?
In future there can be an addition of extra flat file formats and we want to keep the changes to the SSIS packages to he mininum. Just entereing the addiional schema and transformation details in the database should run the package on the new flat file successfully.
Thanks for your time.
Regards,
$wapnil
Hi,
My Problem goes Like this....
I have a folder which contains all the flat files which are used by all the packages(ex--flat file connection managers) in my project.
If we want to change the name of the folder,have to change in every package( in all connection managers) manually.It looks hardcoding and timetaking.
Is there any way to change in one place(xml,file,variable) so that it should be affected in all the packages.
one more doubt is..
If we configure the flat file connection manager in package configurations,configuration file (ex-xml)will be created (we can make changes in that file regarding that connecion mgr only.)
But i want one configuration file (ex--xml) so that i should configure the details of all the connection managers used in all packages.
Hi Guys,
I have a script task that is supposed to read and parse a fixed width source file.
Basically, I want to make the FieldWidths dynamic so that I'll be able to reuse this package with different files. So Instead of hardcoding the field widths directly into my script task, I want it to be stored somewhere that the package can get when executions starts. Is there a way of doing this?
The code looks like this:
Using Reader As New TextFieldParser(mTempFilePAth)
Reader.TextFieldType = FieldType.FixedWidth
Reader.SetFieldWidths(1, 8, 8, 8, 4, 8) <-- I want to change this to handle dynamic widths.
Hello,
I have a huge table of around 6.2 billion rows in Oracle and I need to load this table to SQL Server. I am creating multiple raw files based on contact_dates in the table. I have used conditional expressions to branch out and create raw files. I need to specify static file names for each branch or I have to create multiple variables for each branch and use expressions to create file names.
My intention is to create only 1 variable and evaluate the expression based on the conditional expression description. Is this possible?
Regards
RH
HI,
As the source data of the cube is from MySQL, and the source data volume is more than 80M row per month, I have to build multiple partition in the cube, each partition contain only one month data, even though, the time to load data directly from MySQL is still too long, and because the mysql .Net provider and is not mature enough, the connection often break while loading, so I have to try to load from flat file which was exported from MySQL.
The Partition Processing Destination seems support this way, but, even it shows the partition name in component editor, it actually process the partition with partition ID, and there's not any way to change destination partition name for this component.
Unless I have to change the SSIS package every month, looks like it is impossible to make a smart ETL program that can dynamic create new partition with the ID and name as YYYYMM every month and load data directly from the flat csv file exported from big MySQL table.
Does anyone know how to load data from flat text, and also support dynamic destination partition name?
And I also find a bug in Partition Processing Destination, even 2005 SP2, if there's more than one cube in a SSAS database, and if two partition name in different cube is same, in component Editor, you can not set mapping to the second one with the same name, even you point to the second one, and click ok, the next time you open the editor, you will find it high light at the first one. And even it shows name in editor, it actually process with partition ID instead of partition name, this make it is not possible by change the partition name which need to process to a constant name, say currentMonth to force the component process it.
Thanks.
Jun
I was asked at work to create an audit log to track user changes to our SQL Server 2005 database. My plans were to use a trigger to store the changes in a database table. I was just told that rather then storing these changes the database, that I should write these changes to a text file. I am having problems finding a good example of how to write to a text file in T-SQL. Does anyone have an example of how to do this? Thanks
View 2 Replies View RelatedHello DBA's:
I want to upload 100 text files to a single table on SQL Server 2000.
Records from these text files would be selected on the basis of a where clause.
What would be the best way of accomplishing this? Using DTS, I can do only one file at a time. Is there a faster approach.
Thanks
Vivek
Hi,
I have a table T1 with three Cols.
Code VARCHAR(10),
INISetting TEXT,
TSystem VARCHAR(10).
Right now I are storing the INI Files directly in the database.
I would like to move this to text files in to a SAN box and just store the Pointers in the IniSetting Column and name the text files
some thing like Code.txt
When I do a SELECT Code , INISetting , TSystem FROM T1.
I want to read the Contents as is from the Text files stored in the SAN Box?.
Can some one help me with this.
Thanks
Venu
My problem is simple: i want to dynamic specified the columns in the ContainsTable, this is possible? Please see the example.
Declare @Test1 int
Declare @Test2 int
Declare @Query varchar(50)
Declare @Temp varcahr(50)
--Test
Set @Test1=1
Set @Test1=0
Set @Query='something'
--Add the column to put in containstable
IF (@Test1=1)
Begin
Set @Temp='ID'
End
IF (@Test2=1)
Begin
Set @Temp= @Temp + ',Name'
End
SELECT *
FROM
<table>
INNER JOIN
CONTAINSTABLE (<table>, @Temp, @Query) AS KEY_TBL
ON <table>.ID = KEY_TBL.[KEY]
Thanks
hi allthis is my first post to this group, so pls bear with me while i tryto make some sense.i am trying to create a sproc which uses dynamic sql to target aparticuar table eg. '[dbo].[' + @tableID + '_articles']' and performsome actions.i am using @tableID [int] as a passes parameter for the sproc.everything seems to work fine until i try and manipulate a parameterwhich is of text data type.the error occurs when i try to build the dynamic sql string and appendthe text type variable.eg.CREATE PROCEDURE [procArticlesInsert](@siteID [int],@strShortTitle [varchar](40),@strLongTitle [varchar](60),@strShortContent [text],@strLongContent [text],@intSectionID [int],@intTemplateID [int],@intStatusID [int])ASDECLARE @strSQL varchar (1000)DECLARE @strSiteID varchar (10)SET @strSiteID = CAST(@siteID AS varchar)SET @strSQL = ('INSERT INTO [' + @strSiteID + '_articles] ' +'( [dateEntered], ' +'[shortTitle], ' +'[longTitle], ' +'[shortContent], ' +'[longContent], ' +'[sectionID], ' +'[templateID], ' +'[statusID]) ' +'VALUES ' +'(' + CAST(GETDATE() AS VARCHAR) + ', ' +'''' + @strShortTitle + ''', ' +'''' + @strLongTitle + ''', ' +''''@strShortContent , ' +'@strLongContent , ' +CAST(@intSectionID AS VARCHAR) + ', ' +CAST(@intTemplateID AS VARCHAR) + ', ' +CAST(@intStatusID AS VARCHAR) + ')')GOi could cast the text fields (@strShortContent , @strLongContent) tovarchar, but the restriction of 8000 characters will not go down sonicely.if anyone has any ideas or alternatives to what i am trying toachieve, i would love to hear from you.thanksadrian.
View 5 Replies View RelatedHello,
I have a matrix with a title column. My matrix has a column that is expanded in 2 with data changes
as followings:
MY TITLE1 MY TITLE2
YEAR 2007 YEAR 2008 } matrix
matrix Column column } matrix
How can I change de title (MY TITLE2) differently according to the changing value of the dynamic data column ?
My title2 has to be located above the YEAR 2008 and has to be different from MY TITLE 1.
Thank you for your help.
Best Regards,
Hirondelle.