I want to be able to programmatically set the schema location for an XML source. I first thought it would be a simple task using expressions and variables but it doesn't appear to allow anything in the way of setting it at runtime. Is this possible?
View 4 RepliesDoes anyone know if this is possible:
I have an FTP server that will be receiving files. The directory
and file structure will be a folder with a client name (can be called
anything) and it will have files in it (these files will have the same
filenames as all the other directories. So I will have folder
JimmyDoe with files a.txt, b.txt, c.txt and I will have JonnyDue with
files a.txt, b.txt, and c.txt.
Now I'm trying to figure out a way to get that dynamic file location to
a DTS package so I can import all the data from the text file into a
SQL server. The way the SQL server will be set up is that each
Folder from the FTP site will be a separate Database and each file will
1:1 with a table with the same name..
My biggest issue is figuring out a way to tell the DTS package the file
location to pull all those files and then importing them to the proper
database.
I'm not limiting the solution to DTS packages so if .NET can be
incorporated to make it easier then so be it. But keep in mind I
can have up to 200 folders with 12 - 20 text files ranging from
hundreds of rows of data to many thousands of rows. And the
package needs to be ran twice a day so time/performance is an
issue.
To recap: Need DTS package that uses Dynamic file source and transfers data to Dynamic database destination.
(And I'll write slow VB.NET code to handle this before I create/manage 200+ DTS packages as a solution)
Any help at all is greatly appreciated.
Hi,
Is there a way to set the XSD programatically? I don't want to hard code an XSD filename/location.
Alternatively, is there a way to set the location in a relative path name (i.e. relative to the path of the package)? I did see a similar question here but that wasn't answered. I tried but I wasn't getting anywhere.
As far as I can see there is no way to set this with an Expression either.
Another alternative I thought of would be to set this with using the SSIS object model but I have no experience with this and I don't know if it can be done at run time. If it can be done I would appreciate a pointer to some further information on doing this.
Thanks,
Mark
Hi,
I have placed one table object to show my datarows in reporting services report.
Depending upon the Parameter i want to change the location of this table in report. Is it possible or any other way we can do this. please help on this.
The purpose is to show the table horizontally centered after hiding the visibility of some column depending upon the parameter.
Thanks in advance.
With regards,
Sathish Raja S
Hi,
I have developed a SSIS package on my desktop and the package involves loading of XML data into a database. The XML does not have inline schema and I generated the XSD file from SSIS.
I used Derived Column and Data Conversion to load the data into the database.
Now, i want to migrate the package to a server. Now, when i change the path of the XML and XSD files, all the tasks show error such as
"Input column 'Last_Updated' (4433) has lineage id 3586 that was not previously used in data flow...."
Why is this so? I am using the same XML/XSD files after i moved to the server.
Please advice a fix for this?
Regards,
Vikram
I have a flat source file(.csv) which I am importing to SQL server. Now, if the source file is not available at the specified location, then the SSIS package should retry to execute n times( say 3 times) after certain time interval. The number of retries and the time interval should be configurable.
I have used Flat File connection manager for the source and OLEDB connection manager for the destination.
I am quite new to SSIS. Any help would be highly appreciated.
I have a report that has a matrix. That matrix can have from 2 to 16 columns dependinging on the dataset result. Right now I am forced to place this matrix on the left side of the report and make a column layout pretty narrow. When dataset has more than 13 or so columns it looks OK, but when dataset has only two or three columns it looks weird with a matrix sitting in the left corner with two or three narrow columns and a lot of empty space to the right.
Is it possible programmatically change the width of the columns depending on their number in the dataset?
Is it possible to move the location of the matrix (horizontally) depending on the number of columns in the dataset?
Thank you,
Simon.
I am using the "XML Source Adapter" in an SSIS package.
I notice that although you can specify the XML filename as an expression, the XSD appears to have to be a fixed file path.
This is a problem for me since the path for the XSD is different in my development than it will be in production (in production it's on drive E:, which I don't have).
I'd like to have the file location specified in the config file, but since I can't make it an expression how can I do that?
Also, since they don't have Connection Managers I can't switch DelayValidation on.
Thanks.
Hi All,
I created a SSIS package to extract data from a flat file source and load them into a table in a data base. After I created the package i checked it in to source control(perforce).
But the problem is once a month new flat source file comes and data should be updated.
Once the new flat file comes, is there anyway that SSIS package can identify the path of the flat file and execute the package automatically? In Flat file source only the data will be changed. Not location or data type or anything.
Can i use parameters to do that?
Thanks
I have a requirment which i have partly accomplished , but could not get through completely
i have a file which comes in a standard format ending with date and seq number ,
suppose , the file name is abc_yyyymmdd_01 , for first copy , if it is copied more then once the sequence number changes to 02 and 03 and keep going on .
then i need to transform those in to new file comma delimited destination file with a name abc_yyyymmdd,txt and others counting file counting record abc_count_yyyymmdd.txt. and move it to a designated folder. and the source file is then moved to archived folder
what i have taken apprach is
script task select source file --------------------> data flow task------------------------------------------> script task to destination file
dataflow task -------------------------> does count and copy in delimited format
what is happening here is i can accomlish a regular source file convert it to delimited destination file --------> and move it to destination folder with script task .
but cannot work the dynamic pick of a source file.
please advise with your comments or solution you have
I'm using the XML Source to process a document with 17 elements in it. That leaves me with 17 outputs from the source.
Any time I make even the slightest change to the schema, it causes the metadata collection of the source to be updated. This causes everything in the data flow to have inconsistent metadata. This can take a very long time to fix, even though all I have to do is open the top transform and allow name mapping to work.
Is there any better way to make changes to an XML schema being used this way? Any tricks?
I have a serialized XML that I got from a dataset. In my 'Data Flow Task', I bind the 'XML Source' source to this XML file. Since the XML file is having the schema along with it, I check the 'Use Inline schema' option. However, when I put a dataviewer to see the rows getting sent to the destination, I see that no rows are getting transfered. As you will see from the XML file I am trying to use, I do have one row to transfer.
I tried kepping the schema file and the content file separate and that worked. I am not sure if there are any inherent issues I need to take care of, when using inline schemas to transfer data. I have the SP2 for SQL 2005 installed.
Here is the XML file content:
<?xml version="1.0"?>
<DataSet>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="NewDataSet">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Table1">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="LastName" type="xs:string" />
<xs:element minOccurs="0" name="FirstName" type="xs:string" />
<xs:element minOccurs="0" name="Descr" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
<NewDataSet>
<Table1>
<LastName>Agrawal</LastName>
<FirstName>Sumeet</FirstName>
<Descr>Consultant 1</Descr>
</Table1>
</NewDataSet>
</DataSet>
Thanks,
Sumeet
Hello - Have a project where we are going to build a form creation application. (ASP.NET). This will allow an administrator to build a form on the fly - this form will appear on the front end of the site.This is a fairly common thing. Are there any resources out there as to where to start designing the DB schema? I'm not looking to reinvent the wheel. Here's the basic objects I'm seeing:TablesFormsFormFieldNamesFormFieldTypesFormFieldJSAny tips on the right direction to go?ThanksRob
View 4 Replies View RelatedGreetings!!
I have a MsAccess db containing a table called Employees which i am transforming to Sql server 2005. Everything is working fine. I am using Foreach File enumerator and uploading the files one by one.
However I now plan to validate the schema of MsAccess before uploading it. For eg: My employee table in msaccess is as follows :
Employees
empId int,
empName varchar(60),
empAge int
Since the files come from different vendor, while looping, i want to perform a check if the empid or empAge are not of type long or are not null. If they are of type smallint,i have no problem.
However if they are larger datatypes than the the ones kept in Sql server, then the file needs to be logged in the db with the reason and moved to the error folder. In short, if the datatypes in access tables are smaller than those in Sqlserver, allow it, otherwise reject it.
THe schema of Sqlserver table is same as of that of Employees in msaccess.
How do I do it.
Thanks ,
Lolsron
Greetings!!
I have a MsAccess db containing a table called Employees which i am transforming to a staging table in Sql server 2005. Everything is working fine. I am using Foreach File enumerator and uploading the files one by one.However I now plan to validate the schema of MsAccess before uploading it. For eg: My employee table in msaccess is as follows :
Code Snippet
Employees
empId int,
empName varchar(60),
empAge int
Since the files come from different vendor, while looping, i want to perform a check if the empid or empAge are not of type long or string etc. If they are of type smallint,i have no problem.
However if they are larger datatypes than the the ones kept in Sql server, then the file needs to be logged in the db with the reason and moved to the error folder. In short, if the datatypes in access tables are smaller than those in Sqlserver, allow it, otherwise reject it. THe schema of Sqlserver table is same as of that of Employees in msaccess.
I want to compare the schema of the incoming access tbl fields with my desired schema and all mdb's having data types that are higher or incompatible with the desired schema should be moved to the error.
How do I do it.
Thanks ,
Ron
Is there a way to configure the XML source component to use a schema file whose filename is in a variable? Now I know it doesn't make any sense to have an XML source whose schema can vary, however, I have a parent package with a ton of child packages, all of which process the same XML input using the same schema. What I would like to do is for the parent to load the filename of the XML input file into a variable, and the filename of the XML schema file into another. Each child would have its XML source configured to pickup these variables from the parent, so that if the location or name of the schema changes, I don't have to edit each package one-by-one. However, it doesn't seem possible to specify anything but a filename for the schema. Any suggestions?
View 5 Replies View RelatedHi all,
I am doing an e-commerce project. This website will have a category of product. Each category will have sub-category. And sub-category may have another level of sub-category. This means the number of sub-category is not fixed. In the sub-category we will have products. Each product will have unpredicable number of properties (1, 2, 3 or many properties).
With the current requirment, I can know exactly the number of sub-category level and the number of properties. But the problem comes when later my boss add more category, sub-category ,... sub-category(more sub-category level), and product, as well as the products properties. At that time my database schema will not suitable for new category, products because the in can only design database with fixed number of sub-category level and fixed number of product properties(attributes or fiels in database).
Therefore, I want to ask all of you that
- Is there anyway to solve this problem?
- how to design database that meet extended requirements as I present above?
Thanks for all of your advices.
Quan.
This works in SQL Server 2005:CREATE XML SCHEMA COLLECTION Version2_1 AS'<xs:schema xmlns="http://www.icpsr.umich.edu/DDI" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:doc="http://www.icpsr.umich.edu/doc" targetNamespace="http://www.icpsr.umich.edu/DDI" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:annotation> <xs:documentation> This is a w3c Schema "Technical Implementation" of the DDI Conceptual Specification. This schema is intended for use in producing electronic versions of codebooks for quantitative social science data. CVS Revision information: $Header: /cvsroot/ddi-alliance/ddi/w3c/Version2-1.xsd,v 1.10 2007/07/31 19:03:54 mdiggory Exp $ </xs:documentation> </xs:annotation></xs:schema>' This version, however, gives errors: DECLARE @sql VARCHAR(max)SET @sql ='CREATE XML SCHEMA COLLECTION Version2_1 AS <xs:schema xmlns="http://www.icpsr.umich.edu/DDI" xmlns:xs="http://www.w3.org/2001/XMLSchema"xmlns:doc="http://www.icpsr.umich.edu/doc" targetNamespace="http://www.icpsr.umich.edu/DDI"elementFormDefault="qualified" attributeFormDefault="unqualified"><xs:annotation> <xs:documentation> This is a w3c Schema "Technical Implementation" of the DDI Conceptual Specification. This schema is intended for use in producing electronic versions of codebooks for quantitative social science data. CVS Revision information: $Header: /cvsroot/ddi-alliance/ddi/w3c/Version2-1.xsd,v 1.10 2007/07/31 19:03:54 mdiggory Exp $ </xs:documentation></xs:annotation></xs:schema>'EXEC (@sql)Ultimately, I am trying to create a stored procedure that will take as parameters the name of the new schema and the .xsd file it comes from, so that I can use it in code. I am a novice, so all explanations would be greatly appreciated.Thanks,Ed Graham
View 3 Replies View RelatedI'm having a problem with the XML Source data flow component not transferring the length attributes from an XML Schema to the column attributes of the output table.
An example schema that I have is:
<?xml version='1.0' encoding='UTF-8'?><data xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'><xsd:schema> <xsd:simpleType name='NameType'> <xsd:restriction base='xsd:string'> <xsd:minLength value='0'/> <xsd:maxLength value='50'/> </xsd:restriction> </xsd:simpleType> <xsd:element name='Name' type='NameType' nillable='true'/> <xsd:simpleType name='FamilyType'> <xsd:restriction base='xsd:string'> <xsd:minLength value='0'/> <xsd:maxLength value='50'/> </xsd:restriction> </xsd:simpleType> <xsd:element name='Family' type='FamilyType' nillable='true'/> <xsd:element name='row'> <xsd:complexType> <xsd:sequence> <xsd:element ref='Name'/> <xsd:element ref='Family'/> </xsd:sequence> </xsd:complexType> </xsd:element> <xsd:element name='data'> <xsd:complexType> <xsd:sequence> <xsd:element ref='row' maxOccurs='unbounded'/> </xsd:sequence> </xsd:complexType> </xsd:element></xsd:schema> <!-- data follows --> <row><Name>Fred</Name><Family>Jones</Family></row></data>
When I reference file in the XML Source data control, it correctly infers that there are two columns, but the length of the strings in the columns are set as 255.
This behaviour appears to be at odds with the SSIS documentation (SQL Server Integration Services/Integration Services Object and Concepts/Data Flow Elements/Integration Services Sources/XML Source), which states (highlighting mine):
When the data is extracted from the XML data file, it is converted to an Integration Services data type. The XSD or inline schema may specify the data type for elements, but if it does not, the XML Source Editor dialog box assigns the Unicode string data type (DT_WSTR) to the column in the output that contains the element, and sets the column length to 255 characters. If the schema specifies the maximum length of an element, the length of output column is set to this value. If the maximum length is greater than the length supported by the Integration Services data type to which the element is converted, then the data is truncated to the maximum length of the data type. For example, if a string has a length of 5000, it is truncated to 4000 characters because the maximum length of the DT_WSTR data type is 4000 characters; likewise, byte data is truncated to 8000 characters, the maximum length of the DT_BYTES data type. If the schema specifies no maximum length, the default length of columns with either data type is set to 255. Data truncation in the XML source is handled the same way as truncation in other data flow components. For more information, see Handling Errors in Data.
Has anyone had any luck in getting string lengths automatically extracted from an XML document? If so, where I am going wrong?
Regards,
Michael
How can I get table name and schema names of the source and destination in ssis package to insert into audit table....??
View 3 Replies View RelatedI have a source file name which changes every day , basically the filename like 9500 , the next file would come is 9501 , then next could be anything may be 1000Q. I need the source connection to pick dynamicallly those file name. is their any way i could achieve this in SSIS.
I would not know what is the name of the file i would expect
please advise
I'm looking for a way to transform the contents of n source tablesinto a single destination table. This by itself is no problem.However, the name of the source tables change, so I'll need to basethe transform task on a global variable that I can update via externalcode. Not sure how to do that. I'm ok with executing the package 10times if there's 10 source tables.The last unknown piece is modifying the query used for the transform.There are 10 columns in the source table, but there are 12 columns inthe destination table. I must provide the 2 missing columns. They willsimply contain a year and month, ie. 05 2003.I'm taking a bunch of source tables (for a given month and year) androlling them together into one destination table, and carrying overthe month and year. I assume the month and year would also be globals.But I'm not sure how to incorporate them into the transform task sinceit wants strict SQL syntax.Any help is appreciated, thanks in advance!
View 2 Replies View RelatedHi,
I am building SSIS for 3 different files that have identical
schema and mapping logic.
In my OLE DB Source (object name - "OLEDBSource_SourceTable")
Data Access mode is "Variable name".
As soon as I swithced to this Data Acces mode
it started to give me an error:
[OLEDBSource_SourceTable [1]] Warning: The external metadata column collection is out of synchronization
with the data source columns.
The column "DEAL_NUM" needs to be updated in the external metadata column collection.
The "external metadata column "DEAL_NUM_Flag" (34529)" needs to be removed
from the external metadata column collection.
The "external metadata column "recordID" (33740)"
needs to be removed from the external metadata column collection.
Sorry if this is a pretty basic question.. but my team here is working on SQL Reporting app for the first time.
View 19 Replies View RelatedHi All,
I have a requirement to create a dynamic SQL Command in an OLE DB Source due to the fact that I need to read data from another database based on a date range. For example, the SQL command would look like
SELECT * FROM Table1 WHERE DateField BETWEEN '17/03/2008' and '21/03/2008'
and I need to change the dates - '17/03/2008' and '21/03/2008' to different dates when the package is deployed in production, how do I do that ?
Regards
Ash
Hi,
I am building a data warehouse for a customer who has systems located in two different countries.
I need to import that data from four seperate databases, which all share the same structure.
To do this i have created 20 packages to import that data from the source database. What i would like to do, is at run time set which database the SSIS package should get its data from.
In sql 2k this was easy with a global variable that was set, then use a dynamic properties task to set the data source.
How can i achieve the same result in SSIS? the data source is an ODBC connection, with the four ODBC connections having similar names, eg ABC_NZ, ABC_AU
Thanks in Advance!
Truby
Dear Friends,
I currently have my excel source dynamic for the connectionstring. Using a global variable with the filename and using the expressions of this control to dynamically have the connecttionstring. My problem is that i need tto read always the first workseet and the name changes frequently and generates an erro in SSIS.
how can I set the excel source to goes only for the first sheet independently of the worksheet name?!
Regards!!
Hello,
What I'm trying to accomplish is to have a variable names "SourceTable" and "DestinationTable". So for each SourceTable, the DestinationTable will have the same columns. All I need is to auto-map these columns between source and destination via code?
Is this possible?
Thanks,
awiora
I have CSV file as source for SSIS package every time the filename will be changing like trd_1990M1_1990M12.csv,trd_1991M1_1991M12.csv , trd_1992M1_1992M12.csv etc.,
so it will vary as per user selection . i need to run the same SSIS package to execute the different file name with the same structure.
Please let me know the solution for that how to pass the file name dynamically to SSIS package.
I'm using SSIS to import data from a table (SQL) containing varchar fields. The problem is, that those varchar fields are changing over time (sometimes shrinking and sometimes expanding). I.e. from varchar(16) to varchar(20).
When I create my SSIS package, the package seem to store information about the length of each source-field. At runtime, if the field-length is larger then what the package expects an error is thown.
Is there anyway around this problem?
Oh, yeah... My destination fields are a lot wider then the source fields, so the problem is not that the varchar values doesn't fit in my destination table, but that the package expects the source to be smaller...
Regards Andreas
hi everyone,
joy mundy alluded in her webcast that it is possible to dynamically specify a table name in a parameterized ole db source query. is this true? if so, how can it be done?
Hello,
I have an OLEDB Source component with a Oracle OLEDB connection manager.
In my SQL statement I must do something like this ...
SELECT * FROM OracleTable
WHERE convert(datetime, OracleDate) = parameter
I've dynamicaly build the statement like this : http://blogs.conchango.com/jamiethomson/archive/2005/12/09/2480.aspx
Unfortunately this doesn't work because the variable is a datetime value and not a string.
Maybe another way is to store the date in a table in my SQL database and include it in my Oracle SQL statement ? This would mean that there are multiple connection managers being used in one SQL statement ... I have no idea if this would be possible ...
Any other options ?
I have created an SSIS package that is designed to move data from SQL Server 2000 to an Access db. I have set the package up to accept four parameters. They are:
the name of the SQL Server Database, which is used in an expression to provide the source connection manager connection string;
the full path to the Access db destination database which is used like #1 above.
the SELECT statement used in an OLE DB Source object to get the data from the source table
the table name which is used by an OLE DB Destination object.
I know that the source and destination tables have exactly the same structure and do not require a transformation.
In order to change variables 3 and 4 from above and have it work, I go through the following steps:
I change the variables to appropriate values.
Go to the Advanced Editor for the OLE DB Source object and click on "Refresh". This produces an error in the OLE DB Destination object that is something like "Validation error. Data Flow Task: DTS.Pipeline: input column "strRECTYPESUFFIX" (301) has lineage ID 17 that was not previously used in the Data Flow task.
Go to the Advanced Editor for the OLE DB Destination which brings up the "Restore Invalid Column Reference Editor". I mark all the columns that show up with the option to <Delete invalid column reference> and click OK
I then reopen the Advanced Editor for the OLE DB Destination, go to the 'Column Mappings' tab click 'Refresh', then in the upper pane where the input and output columns list appear I right click and choose "Map Items by Matching Names"
At this point I no longer have the error and the package will execute without any problems.
I am doing this so that I can load the SSIS package in VB.NET (2.0) so that I can then set these variables programmatically and then execute the package. The problem is that this actually performs steps 1 and 4 above. 2 and 3 are left out and the package fails miserably.
I have found some information that would be helpful if I could get my hands on the appropriate object. I realize that the data flow component would return VS_NEEDSNEWMETADATA from the Validate method, and that could be repaired with the ReinitializeMetaData method of the data flow component ( I assume that is the object to be using in this case). But I do not know how to grab the "Data Flow Component" as an object based upon the "Package" object I have loaded so that I can check if it is valid and manipulate it if necessary.
Thank you in advance for your help.