Persisting A SqlDataReader In A SqlFunction Enumerator
Mar 15, 2007
I have built a Table Valued SqlFunction which streams out filtered results from a table. The way that I have implemented this is by using a custom class which implements the IEnumerator interface.
The class internally stores a SqlDataReader and a SqlConnection as private member variables. The Class initializer (ie: the New function) creates a SqlCommand which is executed with ExecuteReader into the SqlDataReader and returns.
The IEnumerator.MoveNext method loops through the SqlDataReader until it finds a result which matches the heuristic filter, and the IEnumerator.Current method returns the current result from the SqlDataReader.
Unfortunately as soon as the initializer returns the SqlDataReader is automatically closed, and I can't figure out why. I've debugged through this and at the end of the Initializer the SqlDataReader is definately open, and as soon as first call to MoveNext is run it is closed.
Can anyone offer any suggestions on how I can fix this.
In the interim i've had to load all of the filtered results into a temporary ArrayList in the Initializer, which defeats the purpose of streaming out the results.
Ever since installing SQL Server 2005 SP1, when using the Foreach Loop container in SSIS the Foreach File Enumerator and Foreach Item Enumerator are missing from the Enumerator drop down box. Anyone else seen this issue? and know how to fix it?
I am trying to use System.IO.Compression to compress/decompress the buffer. I have the data in byte format in sql server and I need to create an CLR user defined function. I have everything running except that for some of the files, the compressed size is larger than the actual file size. I tried to use both Deflate and GZip compression types. Did anyone face the similar situation. Thanks in advance for your help.
Hi I need to have a second copy of our databse on another server as backup if something will happen to the first one. I couldnīt get the sql replication or database mirror to work ( I think the problem i that both servers are running in a workgroup an dnot in a domaine) and now I try to solved my problem with an automatic restore of the full backup (a new full backup is running every night) but now I need a function to check if todays date is the same as a part of the backupfilename and is it possible to do that as a step in the sqlagentjob i have create?
When I try to call a stored procedure by using one of the two marked code segments instead of concatenating string in GetRelation, I can deploy the project to my SQL server without problem. But when I call GetRelation from Query Analyzer, I get an error: "Invalid use of side-effecting or time-dependent operator in 'SET ON/OFF' within a function."
What does the error message mean? How could I correct it? Thanks for answering.
I have a pocket pc app that is using a sql ce (.sdf) database. The inserts, updates and deletes are not persisting when I exit debug. It seems like it's getting put in the database because I bind drop down lists after the inserts, updates, etc and the new/changed data shows correctly after rebinding the data (when I repopulate the drop downs I make a call to the database, I don't just manually add the new item to the drop down so it seems like the changes are getting to the database). However, once I stop debugging and restart debugging all my changes are gone. I'm not using transactions but is there something that I need to do to commit the db changes?
I was able to write successfully a Script Task to set the values of several package variables. But when the execution completes, the variables still contain values which were specified by default.
What I want is a method of persisting the values assigned to a variable through a script. I believe this was possible in DTS
Hi all,I am rather new to database design and modelling concepts in generaland was hoping for some advice on a problem I am trying to solve. Ihave designed a piece of software that creates a tree with pluggablenodes. Each node class can have 0 to n distinct classes plugged intoit to define the type for that node.For example, a node plugged with a 'customer' class and an 'engineer'class would indicate that this node in the tree is an engineer who isalso a customer. We could also have a 'owner', 'engineer' etc.I now want to persist this tree in an SQL Server 2000 Database. I havechosen to implement the nested set model, and have thought about thefollowing table design:table NODE_TABLE:lft INTEGERrft INTEGERpropsID INTEGERtable PROPERTIES_TABLE:propsID INTEGERtableName VARCHARtable CUSTOMER_TABLE:propsID INTEGERfirstname CHARlastname CHARtable ENGINEER_TABLE:propsID INTEGERnum_completed_projects INTEGERdegree CHARschool CHARtable OWNER_TABLE:propsID INTEGERcompanyName CHARSo, given the above example - I would have a NODE_TABLE that links to 2entries in PROPERTIES_TABLE. One entry would link to an entry in theCUSTOMER_TABLE, the other to an entry in ENGINEER_TABLE.Are there any more efficient solutions to this problem? As i said, Iam very new to DB design and would welcome any feedback or suggestionsof how else I might model my pluggable tree in a Database. Thank you,Bob Yohan
I've recently started a new position and our production box. Contains a procedure that uses 30 + temp tables. I'm currently not in a position to change this as it's production and I would have to be granted a window to re-design.
However the tempDb is showing some strange activity.
If a table is created #CarrierService (CarrierServiceID,DeliveryZoneID,CollectionZoneID) for example
Once the procedure is called It will appear in the tempDB with the session info appended as expected
However once the session has ended the above table will get dropped and a new one created
#2C78E45A, I now have 7000 of these different Tables in the TempDB
When I Interrogate this using
SELECT o.name, o.create_date,o.modify_date , c.Name,C.Column_Id FROM tempdb.sys.Objects o Inner join tempdb.Sys.Columns c ON o.object_id =c.Object_ID WHERE o.type ='U'
I get the Following results
name create_date modify_date object_id name #2C78E45A26/04/2015 18:0930/04/2015 14:55746120282CarrierServiceId #2C78E45A26/04/2015 18:0930/04/2015 14:55746120282CollectionZoneId #2C78E45A26/04/2015 18:0930/04/2015 14:55746120282DeliveryZoneId
Push data from the Integration Services pipeline or a custom application. The data can flow directly into an Analysis Services partition from the Integration Services package pipeline, without intermediate storage. This scenario can be used to reduce the latency (and storage cost) of analytical data.
What I think this means is that using SSIS I could bypass porting the data from ODS to DW, and instead use SSIS control flow/data flow components to direct the data straight to the Analysis Services Cube database storage partition.
I was hoping by posting this thread on this MSDN forum I could get details or links from those more knowledeable than I on this subject pointing me to how I may go about using SSIS to implement this approach. Perhaps package code examples or instructions about the components I could use to do this?
I am seeing a particular problem in the XML Source Editor "Columns" configuration where it is not persisting the "Output name" selection.
Control Flow Tab: 1. I use a "Exec SQL Command" to drop, create, or alter the destination tables in the database that I want to be repository for the inbound XML data. The data types are fairly straightforward.
2. I add a singular "Data Flow"
Data Flow Tab:
1. I add a "XML Source" task, and assign a well-defined XML file. I then use the "Generate XSD" option in the "Connection manager"; and I am fairly satisfied with the generated XSD.
2. I create "OLE DB Destination"
3. I wire the "XML Source" to the "OLE DB Destination". In the "XML Source" in the "Columns".
4. I go to the dropdown list of "Output name" and see the list ordered with the various complex-types that I want to map and transfer to a target table.
For the sake of this report, I select the 5th one down on the list (for which I already have a target table) - let's call this "Mesh"
5. In the "Input Output" dialog, I select the "output" to be the desired 5th item, "Mesh"
6. I check all my mappings so that they map one-to-one ... XML name entries match SQL table destination mapping entries; correct types; correct size
7. Check the metadata and it all looks good.
8. When I hit "Debug" to test the package the failure occurs at the "XML Source". The error report comes back saying that it failed because "field xxx in Contributor was truncated". However, "Contributor" corresponds to the 1st name in the dropdown list presented in "Columns" "Output name:".
If I select return to Step 4, when I open up "Columns" I see that my previous selection of the 5th item on the list named "Mesh" was not persisted, but invariably and no matter how often I select item #5 "Mesh" and save to ensure that selection sticks, it is not persisted.
I hand-edited the .dtsx file and only then was I able to make this stick. However, if I ever re-save the package this non-persistency pops up again.
Am I doing something wrong here or is this a known defect? As I have several dozen XSD mappings that I want to transfer to tables, hand-editing is not something I relish.
Would like to do a file System task and a data flow for each row in a table.
I use a ForEach container with ADO enumerator set to a rowset variable. The rowset contains all rows from a single table. What I want to do is as in pseudo-code below:
for each row in ADO rowset
varDestination=CurrentRow.col_file_name.value
FileSystemTask.Destination=VarDestination
FilesystemTask.Copy(Source to Destination)(copy excel file from fixed folder to multiple variable folders)
DataFlowtask.Destination=varDestination(Write data to an excel destination Next My problem is that I dont know how to access the values of fileds from my current row in the rowset. Request help and pointers to samples. TIA Kar
Can someone help me in using the "For each from variable enumerator".
The work around that I have made so far is
1) Generating a Enumerated Value of File collection using Script Control
2) A For each Collection container that processes a given set of files.
the above two conditions works for me when executed independently. But I am not able to relate these two, so as to give a enumerator input to the "For each from Variable Enumerator"
In "Data Flow", I am attempting to use "XML Source" to read a XML document into a recordset and in "Control Flow", into a Foreach Loop Container ( foreach ADO enumerator ), where I map the attributes to DTS variables. SSIS allows you to specify the input columns by means of an index number. The problem is you don't know what index value to use for a particular input attribute. It does not seem to be in the same order as the what the XML input is. This is very troublesome. Ideally, you should be allowed to select the value based on attribute name. Has anyone seen this problem ? Any ideas ?
Is there anyway in XSD to force the parser to return attributes in a specific order ?
I have been trying to find a way to add the File Enumerator Collection option to my version of BIS. Newer versions have a FileEnumerator and I can't seem to find that option. Can anyone help?
I set up a basic ForEach enumerator loop and specified files of type *.sql.
In the directory, I had some files I needed to keep but didn't want the package to touch, so I changed the extension to *.sqlo. Much to my dismay, the ForEach loop picked up those files. (Though it did skip the *.xml and *.bat and a few other types...)
im using a Foreach ADO Enumerator in my SSIS Package, which iterate on a DataSet from a SQL Task.
I use the "Variable Mappings" on my Foreach Loop to retrieve the values into User-variables. But what i need is a way to combine a value from the DataSet with a User-variable and assign this new value to a new User-variable.
I have a client application that creates a COM object using the program ID 'SQLOLEDB enumerator' to get an ISourcesRowset. Once I retrieve the ISourcesRowset, I call ISourcesRowset.GetSourcesRowset to obtain a list of available servers.
This works if the client computer (the one running the application) also has SQL Server (or SQL Express) installed. But on a clean computer after installing SQLNCLI.MSI, the call to GetSourcesRowset returns an error 80004005.
My question is, what do I need to install on my client's machines so I can retrieve available servers using this method? Is there another MSI that needs to be installed? Do I need to install SQL Express Client_Components? Or is there something else?
Thanks in advance for any information you can provide.
I used the SSIS tutorial just fine with no problems. But then I installed SP1 for SQL2005 and now in the Collection screen for a Foreach Loop Container I no longer have an option for Foreach File Enumerator in the drop down for enumerators. Any suggestions or fixes?
I have a simple ForEach SMO Enumerator that returns the names of all the databases in the server identified by the SMO connection manager. I would like to populate the ForEach collection with just a subset of the names, based on some simple pattern matching. In his helpful book Integration Services, Kirk Haselden indicates that this can be done directly in the Enumerate field by adding qualifiers. However, I'm at a loss in regards to syntax.
What I have is: SMOEnumObj[@Name='Databases']/SMOEnumType[@Name='Names'] which returns the database names
What I want is to limit the names returned to those where Left(DBName,5) == "abcdb" (for example).
I have tried adding some test expressions to the Enumerate field, however, the URN is constructed based on the properties of a combination of the Connection Manager and Enumerate fields, and I'm just created invalid URNs.
Hi: I a SSIS beginner and am hoping this is an easy one. I am using a Foreach loop container with an in-memory ADO recordset as the enumerator. The recordset has 3 columns and one of them I wish to fixup data in by calling a web service task that returns an integer. I see how to call the web service and return the integer into a variable but am unsure how to approach putting that variable back into the in-memory ADO recordset? Should I even be trying to do it this way? thanks! Henry
Is there a way to register a new ForEach Enumerator within SSIS, for example ForEach Email in Exchange Inbox?
Or is the solution to create an SSIS variable and use the ForEach From Variable Enumerator? I take it the variable has to implement IEnumerable, or IEnumerable<T>.
My preference out of the two would be to register a new foreach enumerator which would take some implementation away from the developers and reduce time spent on the tasks at hand.
I am using Sql server 2012. In my project whenever I run the Package individually, it run successfully. But while executing the package through SSIS task, I get the following warning and not able to transfer the data from flat file to DB.
Foreach Loop Container:Warning: The For Each File enumerator is empty. The For Each File enumerator did not find any files that matched the file pattern, or the specified directory was empty.
I have a SSIS package this set to run at a specific time each day. If there are no files for the ForEach tool to work upon...while it doesn't 'fail'...I would like to test for the condition that the enumerator was empty...so that I could send an email message reminding someone to followup and investigate.
What would be the best way to test for that condition?
I have a Foreeach loop container that is looking for all the files in a specified directory. For each file in the directory I need to open the file, extract the contents, and pass the contents as a variable for a stored procedure call.
I was able to loop thru the files in the directory and see each file and store the name to a variable.
I was able to use an XML task to open one file via a file connection that connects to a specified file and store the contents in a variable and then execute the SQL task using the variable.
What I am unable to figure out is how to put the two processes together and get the XML task to open the file by using the variable that contains the filename. If anyone could help I would really appreciate it.
I have a ForEach file enumerator that executes a data flow for every source file in a specified directory. If there are no files in the directory, the package executes with a result code of 0.
I need the package to fail if there aren't any files found in the specified load directory. Is there a simple way to do this with the SSIS component, or will I have to script the entire file enumeration process inside script tasks to handle this requirement?
Thanks in advance for any assistance with this question. Please post replies to this forum.
