Integration Services :: Write Key Value Of Records Processed To SSIS Log?
May 5, 2015
What's the best way to write key values of records processed in my SSIS 2012 package to the log provider chosen?My SSIS package deactivates widgets as well as thingies. It was just released into production this week, runs daily, and we'd like to keep a close eye on what it's doing for a couple of weeks, by that I mean on any day be able to quickly see which thingies and widgets were deactivated that morning. It typically deactivates less than 5 widgets and thingies per day.
We could dig through the database to see which were deactivated, but that only works if somebody hasn't manually reactivated it since it was deactivated. We need a log. This is a temporary watch we're doing, so we don't want to write to a table or make make any significant package changes, such as adding new tasks.It seems like writing the 5-or-so deactivated thingy and widget key values to the log is the best way to watch this package. What's the most efficient way to do this? I'm hoping to avoid a new loop and script component with "Dts.Log" calls, but I don't know any other way.
I have 3 columns in my SharePoint list. I wasn’t to select the max date for a particular condition. How to write it in CAML?So from the below table I want to get the max date for column 1 when the value is A or B
I have an Integration services application that retrieves big strings (over 32k) from an ODBC data source and stores them in a SQL server 2005 database as ntext. Somehow the strings are truncated to 32K. The app only copies the data from the ODBC datasource to the (OLEDB) target. Also, some of the special characters in the input text are not properly translated.
I have one ssis package moving the data from staging to destination. In stating table we have the duplicate data. But in destination table 4 columns have primary key. How to handle the duplicate records in oldedb source.
I have an SSIS package that is creating an Excel worksheet and writing data to it. It works fine when i run it inside Visual Studio. But when it runs as a scheduled job it writes the header and no data. I turned on logging and the log even says it is writing the 10,456 rows that it should be.
But they are not showing up in the Excel document. The job is setup as 32 bit and writing to Excel 97-2003. The job ends normally and does not generate any type of messages that are out of the ordinary. This is running on SQL Server 2008r2.
I'm looking at SSIS and SqlBulkCopy as a possible method to quickly insert and process large amounts of data, my current method uses the sp_xml_preparedocument and OPENXML to parse an xml document of the data I want to process and insert into the database, however I'm noticing the performance of SqlServer parsing the xml document isn't that good.
I found the C# SqlBulkCopy method (new in .NET 2.0) and I was thinking it would be faster to use that to load my data into a staging table and then use SSIS to extract the data from the staging table, process and transform it as necessary and finally load it into the final destination tables. I was able to create an Integration Services Project that selects the data from the staging table, does a bit of processing on one of the fields (by calling a stored procedure), and finally loads the processed data into the final table.
The problem with this is I need to clean out the rows that were extracted from the staging table and I'm not sure how I can accomplish this (and I can't just issue a "delete from staging_table" because there maybe new records in the staging table that were not processed), perhaps I can either delete each record as it is proccessed or somehow get the last proccessed identity id from the staging table and delete all records less than or equal to that id.
thanks in advance for any help you can provide, maybe there is an easy way to accomplish what I'm trying to do.
I am in middle of my transformation where I have to assign records equally among 3 different groups. I can do that in SQL using NTILE() Over() function. How do I do that in SSIS package. I have applied different business rules during transformation to get unique records and now I have to assign those records to 3 group in and generate excel report.Basically, I will need to have another column which will have those group numbers.
I am trying to load a simple Excel file into a Database table and the SSIS Package is not loading any records beyond 3233 records. I am just surprised. I tried using the "IMEX=1" mentioned in some of the online resources but it didn't work. I am using an Excel Source, a Data Conversion Transformation and an OLEDB Destination in my package in SQL Server 2014 (which is pretty simple and straightforward).The Excel file I am trying to load can be found here.
And, here is my table structure.
CREATE TABLE [gov].[loan_limits]( [FIPS_State_Code] [varchar](3) NOT NULL, [FIPS_County_Code] [varchar](3) NOT NULL, [County_Name] [varchar](50) NOT NULL, [State] [varchar](2) NOT NULL, [CBSA_Number] [varchar](6) NOT NULL,
I have a stored proc that is returning the results I need for output to .txt file.
Is there a way in SSIS to commit 50K (or whatever number) row batches at a time or should I just handle this in the stored proc?
select * into #TempTable from SomeTable [WHILE LOOP] --throttle commit batches of 50K rowcount select * from #TempTable [END LOOP] drop table #TempTable
But If I'm doing this in SSIS, I can't drop the #temp table otherwise I have nothing to output right?
I have a Problem with my destinations. I have a split condition with two ways the flow can use.
In this case: all and Date.
All and Date can be set by using a variable. Its working good.
When a user fills the variable with a date value (cast to string) the conditional split executes the correct flow with all the needed rows... The same time the all flow will be executed with 0 rows. In the end the destionation file for the all values will be overwritten with nothing. The same on the other hand when a user fills the variable with the all value, the date file is empty. What can i do to make sure that the files are not empty?
Suppose in my table i have 300 records. In that 300 records i want to update first 100 records with today's date. 101 to 200 records with yesterday's date and 201 to 300 records with tomorrow's date.
I want to achieve the following in (SSIS/SSDT for SQL 2012) -
I have a generic SSIS package which simply sends out email notifications using SMTP email task (this package is within its own project, and has project level input parameters).
I need to be able to call this package in the Event handler section of every package (numbering in about less than 60) that we have. These packages are within their own respective projects.
I thought I could use the "execute package task", but it turns out , using this, I cannot call a package that is part of some other project. I also cannot call a package that is stored in the CATALOG. Is there any way I can do this ?
When I call the child package , I should be able to send in parameters like - error information and package name of the Parent package.
Now, I want to merge the source data with target table -that means, if the records are already avaible in target, then ignore and if it does not available then INSERT.
This is the query i used but new records are not getting inserted.
MERGE #target T USING #source S ON S.SOURCE=T.Source WHEN NOT MATCHED BY TARGET THEN INSERT ( Source, Prefix ,tgt_patientcode ,tgt_patientdesc) VALUES ('Canada' , 'cn' , s.patientcode, s.patientcode);
I have two records in the source with information ID, RevisionID, Description, Region
There are two lookup files one with ID,Description amd other with ID, Region
I wish to update my two source records with performing lookup with these two files.To get the correct description and region data. How to do this in ssis DFT.
I have a transformation where final result set give me 25 rows of data. Now before I put into destination table, I need to add another column which will show how many total records we have. Like.
My dataset:
A 20 abc B 24 mnp c 44 apq
Now I need to add another column within my transformation before I store the result set to destination like this:
A 20 abc 3 b 24 mnp 3 c 44 apq 3
Here. new column gives count of total rows in our dataset which was 3.
How can I achieve this? Can I use derive column to this?
I have an Excel file which contains some data. I want to load that into a SQL server Table. Here are my conditions :
1. If the table doesn't have any matching records from the Excel file, then my DFT should load the data from that Excel to the Dest Table.
2. If the table has even one or more matching records, then the DFT should not process at all, instead I should send an email to the business stating that there are some matching records and hence the package is not process...ed.
P.S. If i use Lookup, I have two matching and non-matching output. which will process the non matching records into the table and matching can be redirected to any flat/Excel file. But i don't want to do this. I just want to lookup the Sql Server table and excel.
It'll be good if there is an additional option in the Lookup "Fail component on matching records".
I have a scenerio where i need to run the scheduled package for every 10 minutes. Let me give some examples. Say once my ETL process is done, my SS_Batch table gets inserted once. Here i have two columns like SS_Batch_cd and SS_Create_TS respectively. Once the ETL process runs successfully this SS_Batch_CD column will have values as 'C' which means 'Completed'. Similarly when ETL process fails this SS_Batch_CD column will have values as 'F' which means 'Failed'. And Similary when ETL process is in progress this SS_Batch_CD column will have values as 'P' which means it is in 'Progress'. In SS_Create_TS column, it will have date like 2008-04-15'.
Note : This SS_Batch table will get inserted only once when ETL job is over (whether the job is runned sucessfully or not or in Progress) along with the date.
Actually i need to run the package for every 10 minutes because i dont know when the last cube was processed. If i get this last processed cube date then i can check this processed cube date with SS_Create_TS column and SS_Batch_cd in SS_Batch table. Say if Last processed cube date is greater than SS_Create_TS then i can refresh or process the cube. This validation i need to do to achieve my goal.
What are all the control flows should we need to have in SSIS to achieve this scenerio? Please give me briefly to solve this problem.
if there is any way to accurately size a single server using SSIS. The server will be a virtual machine. The data being loaded will be approximately 200 MB per load with loading to a 150 GB database on a separate server.
I have scheduled SSIS package through Sql Agent and when I right click on job start job as step package runs successfully but when I schedule job it dosent run.
Can anyone help regarding the SQL server integration Services(SSIS), ETL We have requirement like this: We have Live Database( LIVE_DB ) and Reports Database (REP_DB) I want to trasfer the few tables data from LIVE_DB into the REP_DB for end of the day using SSIS If any new records are added, updated or deleted in LIVE_DB, these should reflect in the REP_DB, Our requirement is not to delete the old data, we should append or delete or insert the new transaction data in REP_DB.
Thanks in advance, if anyone help me in resolving this issue.
I have a maintenance plan which consist db full backup and log backup ( in two subplans), I execute both on SQL agent, and both failed.The DB Log Backup : DB FULL BACKUP LOG:
Hi All, I have some clarifications on stopping my package once cube is refreshed or processed.
Below i have given steps for the transformation in my package
Let me give you what are all the dataflow transformations that i had given in my package.
1. Data Flow Task
2. Script Task 1- I have written code for getting the last processed cube (global variable has been declared for Last processed cube date - lastProcessedCube)
3. Script Task 2 - I have written code for SS_Batch table where i can get Create_Ts date that is assigned to another global variable - create_ts.
4. Analysis Processing Task.
In between Script Task 2 and Analysis Processing Task i have given @lastProcessedCube > @create_ts for Expression and Constraint under Precedence Constraint Editor
Actually i need to run package for every 10 minutes which i can do it in Job Schedule and need to refresh or process the cube daily. Is there any way to stop the package once when my cube is processed on that day. Again start the package for the next day.... Is it possible to do this? Please let me know.
I have a ssis package with an oledb connection using windows authentication. i want to understand when i promote this package to the server and add it to a job, then a user login to the server with sql server authentication and run this job. Which/what windows authentication this package gonna to use to connect to the server ?
