I need an advice for an easy maintainable and highly flexible solution using SSIS. We're supplying our customers with an exportservice. The data is extracted from different tables, all with the same tableschema, fairly easy to create a foreach-container and iterate through the tablenames and extract data and use a flatfile destination to write it to...
But the not so easy part is to have different exportformats and still be using only one package. I could maybe use different ConnectionManagers for the FlatFileDestination, but that's not a very easy maintainable solution in my world. Our Customers will soon demand a webinterface where they can select necessary columns and apply different formatting for example they would demand the datetime to be '2006-12-24' or maybe '12/24/2006' etc. for alot of the available columns.
Any good suggestion on how to accomplish such a task?
I receive several TXT files daily that need to update information in SQL Server databases. The process requires that all TXT files be appended to a master file and also update individual files' information based on the TXT file name. For example:
File TABLE1_x_ddmmyy.TXT (ddmmyy = date, x = "O" or "B") is to be appended to the master file and also update SQL table "TABLE1" by setting a flag for those records in the table that match a unique key that is provided in the TABLE1.TXT file.
In VFP, I had the following process in place: a) open the TXT file. b) read its file name and open the corresponding VFP file c) update the VFP file based on the key provided in TXT d) append the key to the master file. e) repeat c-d for next record in TXT f) repeat c-e for next TXT file
Using the same process with ADO takes a considerable time since I am processing one line at a time.
Is there any way to do this using a DTS package of some sort? How can I read the TXT file names in SQL Server?
User-B Custom fields: Blood type <char 3> Date of birth <date> Referred by <varchar 50>
Different users can have different custom fields in their address book. As you can see, while the standard fields for each user can be
stored in a single table. However, I have several methods by which I can store the CUSTOM fields.
------------------------------------------------ Method 1: Create 2 separate tables called CustomField and CustomValue:
CustomField has fields: FieldID <int> FieldName <varchar 25> UserID <int>
CustomValue has fields: ValueID <int> Value <varchar 50> FieldID <int>
------------------------------------------------ Method 2: Create a separate Field and multiple Value tables for each data type: CustomField, CustomCharValue, CustomIntValue, CustomMoneyValue, etc...
CustomField has fields: FieldID <int> FieldName <varchar 25> FieldType <smallint> (determines which TABLE, below, contains the data) UserID <int>
I'm thinking that while Method 1 will be easier to implement, Method 2 may offer me better performance if coded correctly. I'm going
to assume that I'll have at least 1-5 million records to work with over the course of my first year and I will need the ability to sort
records based on values in the custom fields as well.
My first question is: Which method should I be considering and is there an alternative or hybrid that I should be considering?
My second question is: What statements should I use in my stored procedure that will enable me to retrieve a list of USERID, CustomFieldIDs and their values as one resulting table that I can query at will and with solid performance?
Hi, I have this idea of using Dynamic SQL on a cross-tab and the following questions popped-up:
1) Is there a limit to the length of the statement the EXEC / sp_executesql command can execute? 2) Should I use cursors (Haven't used this one, I preferred work tables)?
Here are the details:
TABLE_SAMPLE RptDate DATETIME, SomeField CHAR(1), ID CHAR(8), RetVal VARCHAR(15)
SP Input: A date range that defines what the RptDates are: @DateFrom DATETIME, @DateTo DATETIME
[BLANK] - means that there was no record for that RptDate for the ID
Conditions: - The number of records per ID varies and depends on the number of RptDates covered by @DateFrom and @DateTo ID1 can have records for 02/26/2008, 02/25/2008, 02/22/2008 ID2 can have records for one, two or all of the same dates. etc...
I'm trying to create a DTS package that will let me import an Excel file. The user will be able to name the file the same name every time. But can the DTS package read a different worksheet name each time? Right now, if I use the Excel connection object in DTS designer, it wants to hard code the worksheet name.
Is it possible to set up the permissions to not allow a specific user to create schemas, but to allow that user to create tables and procedures and functions in one schema, and to create procedures and functions but not tables, in a different schema within the same database?
Hello All, I have a requirement, where the number of parameters being to a stored procedure, is not fixed. It is to have a list of computers, belonging to a particular Domain, or, more DOMAINS or maybe just a list irrespective of the Domain. For this, the @Domain parameter, could have one value, or more values, or no values as well. Can you please let me know how do I go about this?
(the SP has two parameters; I get the same results with two question marks. The connection is an OLE DB connection.)
I have two input variables mapped to the Execute SQL task, User::company_sysno (GUID) and User:: start_period_sysno (LONG). Depending on the iteration, I also have a returnvalue variable mapped, User::ErrorCode (I have tried every data type I can think of). The stored procedure does contain a RETURN @m_intErrorCode line, but the behavior I'm getting does not change if this line is commented out.
When the ResultSet is set to "none," I get the following error:
"Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done."
When the ResultSet is set to either SingleRow or FullResultSet, I get the following, somewhat different, error:
There is an invalid number of result bindings returned for the ResultSetType: "ResultSetType_SingleRow".
(Obviously, change the last bot of that to "FullResultSet" when FullResultSet is selected.)
I've found a handful of webpages dealing with each of those errors separately, but have not discovered one that deals with them in tandem, so I'm not even sure where to start looking. For that matter, I'm not even sure what else I need to tell you to give you an idea of what's going on. I'll post the stored procedure if necessary, but it's rather long, and my company are a tad paranoid about releasing code into the wild.
Hello! Given three dimensions in my cube (Product, Calendar, & Sales). I'm trying to create a calculated measure that will sum 8 weeks of sales starting from the third week in which sales started. For instance, weeks 1-4 had zero sales. Week 5 is the first appearance of any sales. Therefore, the measure would sum the sales of weeks 7-14 (8 weeks starting with the 3rd week after sales begin).
Any ideas?? Keep in mind that every product is going to be different (sales start in different weeks).
User-B Custom fields: Blood type <char 3> Date of birth <date> Referred by <varchar 50>
Different users can have different custom fields in their address book. As you can see, while the standard fields for each user can be
stored in a single table. However, I have several methods by which I can store the CUSTOM fields.
------------------------------------------------ Method 1: Create 2 separate tables called CustomField and CustomValue:
CustomField has fields: FieldID <int> FieldName <varchar 25> UserID <int>
CustomValue has fields: ValueID <int> Value <varchar 50> FieldID <int>
------------------------------------------------ Method 2: Create a separate Field and multiple Value tables for each data type: CustomField, CustomCharValue, CustomIntValue, CustomMoneyValue, etc...
CustomField has fields: FieldID <int> FieldName <varchar 25> FieldType <smallint> (determines which TABLE, below, contains the data) UserID <int>
I'm thinking that while Method 1 will be easier to implement, Method 2 may offer me better performance if coded correctly. I'm going
to assume that I'll have at least 1-5 million records to work with over the course of my first year and I will need the ability to sort
records based on values in the custom fields as well.
My first question is: Which method should I be considering and is there an alternative or hybrid that I should be considering?
My second question is: What statements should I use in my stored procedure that will enable me to retrieve a list of USERID, CustomFieldIDs and their values as one resulting table that I can query at will and with solid performance?
I have a large SQL 2012 table containing survey details. The number of questions vary in each survey and can range in number from as little as 10 questions to a maximum of 50.I need to adapt my crosstab code below to include a CASE statement that outputs a column (Q1, Q2, Q3 etc) representing the questions up to a maximum 50 questions (Q50) and to place the answer under the corresponding question column within each survey. Ideally I want to avoid having to write 50 CASE statements in my code. I chose the CASE statement method as I understand that the PIVOT option isn't as flexible,I have included some test data and the output should look like this:
True story. Thanks. Dont think that fixed or ragged is going to work for me though. File does not seem to be structured.
I have what appears to be a fixed length file that I would like to import using SSIS. However it does not look loike the spacing between fields is constant. Is there a way to say from character 1 to 10 is X, and 11 to 15 is Y, etc etc? Below is a couple of rows that all pertain to the same record.
01000000225672101242253 55232101242253 5TKTT / 0101 FFVV 5235MCYSNR/CE MUSSETT/BRIAN.A.MR 20071017U7700002277000022 U00000000ZAJNB JNB ZA 00000000 00000000 00000000 SITII Y06 405 02000000235672101242253 55232101242253 5ICE 20071017 1407000017OCT 17OCT 03000000245672101242253 55232101242253 5 PLZPLZ ZAR 1000 ZAR 72ZAZAR 140ZVZAR 562XTZAR 1774 ADT PENALTY APPLIES 04000000255672101242253 55232101242253 50000017740000000100000ZAR200000000000 0000000000000000 0000000000000000 00000000000000000000000000000000 000000000000000000000000000000000ZA 00000007200ZV 00000014000EV 00000001600YR 00000054600 00000000000 00000000000 00000000000 00000000000 00000000000 05000000265672101242253 55232101242253 51 OPLZ DUR CE CE 0633 0633 K K 18OCT18OCT18OCT1625 1740 OK KOW 2 DUR PLZ CE CE 0602 0602 B B 20OCT20OCT20OCT1000 1115 OK BOW 07000000275672101242253 55232101242253 5CCIK5221 000001774005221591015192982 0308M067781 00000000000 00000000000 00000000000 08000000285672101242253 55232101242253 509PLZ CE DUR550CE PLZ450ZAR1000 END CE XT16EV546YR 1 2 3 4
What do you reckon is the best option? SUBSTRING in a Dervied Column? Below is an example of the text.
I'm writing a query that will be used in Jasper Ireports, but prefer to have the values done ahead of time using SQL rather than relying on the report to do the lifting.The fields are pretty straight forward, only the display is where I have a question.
Fields Used: PERIOD ('MON-yyyy') and VALUE
The results must start with the CURRENT PERIOD (AUG-2014) in one column and the VALUE for the current period multiplied by 1/12 (VALUE*(1/12)).The next column should return the VALUE for CURRENT PERIOD - 1 (JUL-2014) and multiply by 2/12 (VALUE*(2/12))
This should continue for the last 11 months and would end with OCT-2013 with the value being multiplied (VALUE*(11/12)).Is the easiest solution to this a CASE statement looking at PERIOD then PERIOD minus one month, minus two months...etc?
I get error reports in simple text files like the one below in relatively the same format. The only thing that varies is the number error reasons as there can be any number of error reasons for a file. Usually there is only one but there can be a handful. What is the best way to capture the error description and count of errors no matter how many there are? I want to take these items and update a table I have in sql server 2008r.
Original File Name: some.file.YYYYMMDD.d.incr.02of02.1.dat Source File ID: file02YYYYMMDD File Receipt Date: 10/17/2014 Total records received: 1331136 Total records loaded: 1329987
ERROR REASONS Error code: EBBW002 Error desc: Duplicate Record Total records: 1146 Error code: EABC001 Error desc: Invalid Length Record Total records: 1 Error code: ERRCM10 Error desc: Missing First Name Total records: 2
Total number of Errors encountered during the ODS update processing: 1149 ************************************************** *****
I need to import data to a MSSql table from massive (read: a million and a half rows, every single day) logs that come in .txt format separated in tabs with a ";" symbol and then have some stored procedures analyze that data to generate some reports in an excel file with that info. The text files include the column headers in the first row and the data starts on the second one.
The challenge is that the text files differ in column order and count every single day.
The analysis that I need to do only needs about 15 columns from the nearly 90-120 that those files include, and those columns sadly happen to be in a different order in those files.
I'm new to programming with the ReportViewer object and this issue has me stumped: it appears if you have some optional parameters in your report, and a way to refresh that report with different parameter values, the report "remembers" parameter values from previous calls to SetParameters() on subsequent renderings of the report. If a parameter is included in a call to ServerReport.SetParameters() on the first rendering, but not included in a subsequent call and the report is re-rendered, the previous value of the parameter (rather than the default value) appears to be used.
Here's a snippet of some test code I wrote within an ASP.NET 2.0 test application:
retVal[paramCount++] = new ReportParameter("Name", txtName.Text);
if (txtAddress.Text != "")
retVal[paramCount++] = new ReportParameter("Address", txtAddress.Text);
if (txtZip.Text != "")
retVal[paramCount++] = new ReportParameter("Zip", txtZip.Text);
return retVal;
}
The test report was written to simply echo back the values of the parameters that are specified. The report definition allows NULL to be specified for the parameters.
The test app was written so if I enter a blank value for Name, Address or Zip, the corresponding parameter does not get created in C# and does not get sent to the report server. If I view the report with all three values (parameters) filled in, I see the parameters echoed back to me in my simple report as expected. If I clear the parameter values the first time the report is rendered, none are sent to the report server and I get no values echoed back in my report, also as expected. I can change the values and click on the View Report button and see the new values for the parameters as expected. However, if I clear any previously-specified parameters and click on View Report, the previously-specified values for the ones that are now cleared are still displayed by the report.
So my question is: once a parameter has been sent to the report, how does one "unsend" it on subsequent refreshes? I know I can create the parameter and set its value to null...but I have a situation here where that can cause errors. It'd be better if I could simply leave out the unspecified parameters and have the report refresh and render as if I were rendering it for the first time.
How do i import a Varying Column Width Flat file into a Table using SSIS?
I have a flat file that has 4 columns with varying width Like I should read the file as Col 1 - (1 to 10 Characters) Col 2 - (12 to 21 Characters) Col 3 - (22 to 35 Characters) Col 4 - (36 to 38 Characters) At the end of the record is a "LF"
I think "Fixed Width" Columns allow me to define a standard column length for all the columns.. Right?
I am new to SSIS. I have been struggling with this for the past one week. I have a weird task. I need to import several tables from one database to a different server with a new database name. We need to do this at the end of every year. The main problem here is that the number of tables varies every year. You may not have all the tables as last year or may have more tables. So I need to create a dynamic task that takes care of this every year without changing the package.
I have performed the following tasks **
1. Create a new dynamic database. ( I have used Execute SQL Task to do this) 2. Copy all the table structures ( I have used Execute SQL Task to do this)
3. Import Data. This is the main problem. I was trying to create a dynamic connection string with variables as suggested in several forums but I finally came to know that this cannot be done if the table structures are different as the metadata cannot be refreshed at runtime.
4. The final step to create a process to validate the data (the count from each table for both source and destination. I think this can be done with Sql task.
What is the best method to do this? My DBA does not like “Transfer SQL Objects Task” or “transfer Database Task”. I would like to create this as a dynamic process.
