Hi,
I have an SSIS package that runs each day from a live data source to create a data mart, which is then used for various things including SSAS and SSRS.
The problem is that certain records that will eventually go on to form fact tables are deleted from the live system (not a very robost database in the first place, hence the SSIS!) but these are not reflected in the SSIS transformation, creating plus figures when compared to the live system.
I currently use type 1 slowly changing dimension processes in each data flow (of which there are about 35) but I realise that this only updates records and does not delete.
The solution I have in place is to truncate the fact tables in the mart before the run starts using an Execute SQL task. This solves the problem though to me seems a little heavy-handed and renders the slowly changing dimension processes redundant (as it is currently only run once a day).
My question is, is there a better method of dealing with the above scenario? If there isn't, it would be a nice feature to add to future versions (*nudge nudge*).
Iam using Sql Server Integration Service to transfer the data. I have to methods to transfer the data.
Method -1
Daily cleaning Dataware house Data base and transfering the data from Source database.
Method -2
Only new rows transfering from the Source database to dataware house data base.
If i use first method, Any performence issues's will come in the future?. In future my source data will have upto 6 lacks records.
If i use second method, daily date based i can transfer the data. But if they delete any previous records from the source database how can i reflect the same in Data ware house Data base.
I want to skip running the SSIS data flow task when the source file is missing. We have a scheduler that copies the source file to the staging area. This SSIS package runs as SQL server job. So when a SSIS package fails due to missing file the remaining steps in the SQL scheduler won't execute. I want to handle the missing source file condition grace fully. Please advise.
Is there a way of stopping truncation warnings due to the fact the the XL driver assumings BSTR(255) columns. I'm loading data from excel to SQL Server, the SQL Server columns are nvarchar(50) but the Excel source has assumed 255.
I can change the Excel Source using the 'Advanced Editor' and change the meta-data of the 'Output Column', but then there's a mismatch between the External Column and Output Column collection - so a warning is displayed.
The External Column meta-data can be changed, but then it's out of sync with the 'Error Output' meta-data (and the Error Output meta-data is read only).
I'm sure I've missed a step, can anyone enlighten me
There's lots of smart people here who can, I am sure, offer insight on this one.
I need to add handling of erroneous data to my package.
This is what my SSIS package does. The source table has thousands of records in it, most of which have been processed already (have a status of "Finished".) A typical run would have 400-600 records to process. The runs happen overnight.
Steps: 1) Execute SQL task that Updates all records in input table with status "New" or "Error" to "Process" 2) Data Flow task that takes as input all records with status "Process" and outputs to destinate OLEDB table 3) Execute SQL task that Updates all records with status "Process" to "Finished"
Erroneous records are identified at step 2 (Data Flow task) and need to be marked as "Error" so that they are not marked as "Finished" by step 3. They will be picked up and reprocessed by step 1 next time around.
I am having trouble seeing what would be the best way to achieve this? I am concerned that any steps I take might be prone to deadlocking as the updated data is also the source data?
My initial thought is to output some data to an "error" table in step 2, and then insert another Execute SQL task before step 3 that will update the source table to mark all these records as "Error" records. Does this make sense? Is there a best-practice way to achieve this?
I'm trying to import a flat file source into a SQL Server table.
The flat file is pipe-delimited and text qualified with " (double-quotes).
The import job is failing because there is a "comments" field in the flat file and there are carriage returns within some records in the "comments" field. When SSIS encounters a record with a carriage return within that field, it sees the carriage return and assumes the end of the record, even though the field is text qualified with " ".
The actual error message I see is: "
Error 0xc0202055: Data Flow Task 1: The column delimiter for column "comments" was not found. (SQL Server Import and Export Wizard)
Sample Record: "418186"|"94"|"Staff Only-Minimum charge out of 3 hours Plus travel & accommodation costs (if required) – at cost.
All trauma response services & associated fees/costs are required to be formally authorised by the Company prior to delivery."|""|"690"|""
I can't think of a way to get SSIS to ignore the carriage returns within the Comments field in the source flat file!
I have a table with about half a million records, each representing a patient in my county.
Each record has a field (RRank) which basically sorts the patients as to how "unwell" they are according to a previously-applied algorithm. The most unwell patient has an RRank of 1, the next-most unwell has RRank=2 etc.
I have just deleted several hundred records (which relate to patients now deceased) from the table, thereby leaving gaps in the RRank sequence. I want to renumber the remaining recs to get rid of the gaps.
I can see what I want to accomplish by using ROW_NUMBER, thus:
SELECT ROW_NUMBER() Over (ORDER BY RRank) as RecNumber, RRank FROM RPL ORDER BY RRank
I see the numbers in the RecNumber column falling behind the RRank as I scan down the results
My question is: How to convert this into an UPDATE statement? I had hoped that I could do something like:
UPDATE RISC_PatientList_TEMP SET RRank = ROW_NUMBER() Over (ORDER BY RRank);
but the system informs that window functions will only work on SELECT (which UPDATE isn't) or ORDER BY (which I can't legally add).
Hi,I used sp_addlinkedserver to link to a remote server through ODBC.When I execute select count(*) from LinkSrv.SI.DBO.SIHeader in SQL QueryAnalyzer. It returns 13705 records. But when I execute select * fromLinkSrv.SI.DBO.SIHeader. It only return 885 records. If I specify somecolumns, select ODCOMP, ODPONO, ODVDCD from LinkSrv.SI.DBO.SIHeader.It returns more records, 1213 records.I guess there is something limit the return storage, but I can notfind it.Any suggestion will be appreciated. Thank you
I am using this code to delete records from a table in the database that has two fields (both are primary keys) I am not getting any error, but nothing gets deleted. Thanks for the help. Code:protected void btnRemove_Click(object sender, EventArgs e) {
I want to know what to do to recover the deleted records on a specific table. Unfortunately, the dba was unable to activate Trace feature and I am not familiar in using the transaction logs to retrieve data.
Does anybody know if SQL 2000 has a tool to roll back (udelete) records? Let's say Payroll dept deletes 500 records by mistake... I know of 2 products that do that (Computer Assoc. - Unicenter Log Analyzer and Red Matrix Techmologies' SQL Audit 1.7 - but I have never used them. Any comments? Thanks for your help.
Is there a way to check who has deleted records from my SQL tables? I asked because I notice records keep disappearing from my tables recently for no reason. The DB is only accessible by a few IT staff. Business users have no direct access to it so they can't do any harm and there is no application that update these record missing tables.
I don't need to know the exact records that have been removed. I need info on who has made a deletion activity on which table, date and time is good enough.
I'm running a DB using MSDE (2000) that is interfaced by 2 differentades running on PCs with Access 2000 Runtime. One of the ADEs is apackage accounting system that is very solid and stable, the other isa custom application that I wrote (much less solid and stable). Thecustom app only deals with a select few tables in the database, andthe table in question is not one of those.With alarming regularity(daily), records are getting deleted out of aparticular table. I've set up a couple of dummy records in the tableand put a delete trigger on the table that creates record in a 'log'table that tells me the user and the time that the records aredeleted.The deletion (all records in the table) always occurs during businesshours (never over the weekend or at night) and the user responsiblevaries among 3 or 4 different users. 2 of those users don't even haverights to that table, so I'm really confused how those logins couldcause a delete on the table they don't have access to!??!As far as I can tell, this is only happening to this particular table( I hope!).Is there a way that I can get more information on the process ormachine or anything else that is behind the deletion?
I am trying to use SSIS to upsize Visual FoxPro data. The data is in free tables. I'm using the VFP OLE DB provider in a connection pointed at a folder. When I click on preview, I see the right data--deleted records are not included. However, when I run the package, the deleted records are sent to SQL Server 2005. I can pack all the tables prior to executing the package as a work around, but I would think I shouldn't have to. I don't see any way to detect which records are marked for deletion in the data flow. I've tried to specify in the connection string Deleted=Yes, but to no good effect. Also, as I mentioned, when I preview the data, the deleted records are not included, so the connection seems to work properly . Steps to reproduce: In VFP:
CREATE TABLE Test (Field1 N(1))
INSERT INTO Test (Field1) VALUES (1)
INSERT INTO Test (Field1) VALUES (2)
GO TOP IN Test
DELETE
In SQL Server:
CREATE TABLE Test (Field1 Numeric(1))
In Visual Studio SSIS:
Create connection object using VFPOleDB provider pointed to the folder with the Test.DBF.
Create a connection object to SQL Server 2005 pointed to the database with the test table.
Create a DataFlow task with the OLE DB Source and the SQL Server destination. When you preview the Source object you will see just one record, but when you execute the package, you see 2 records transferred and 2 records are actually in SQL Server. Seems like a bug.
We occasionally use TruncateTable to delete the contents of a number of tables on our SQL Server 7 database. We then build them anew from an outside source. I am finding that the size of the database (and corresponding backup file) has grown significantly. As I understand it, the space representing the deleted records is retained by the database. I have investigated using various DBCC commands, to no avail. Does anyone have a clue as to how I can recover this lost space? ... Thanking you in advance. Mike Rosen - Amalgamated Bank
Hi I am using sql server 2005 express and would like to keep all my fields from being both updated and deleted.
In other words, once I create a new record, I would like to have it protected from being deleted and I dont want the field values to be updated/changed from the values initially entered. Is there a way to this without running triggers or changing database permissions and user roles?
I tried making the database read-only, but then of course i cant add new records.
It happened accidently that we deleted records from a table in SQL server 2005 DB. We never took a full backup of the DB till then. Is there any way that we can recover the deleted records. Logs files are still present. ( *mdf and *ldf ).
installation of their softwares? If there has NEVER been a backup ofthe Database (.mdf), Recovery Model Set to Full, and there has neverbeen a backup of the Log (.ldf) files?Using the trial software for both, this was unclear.Thank you for your time,bd
I have 2 tables say table1 and table2 with the same structure. Each recordis identified by a field 'SerialNo'. Now there should be a total of 500000records in both tables with serialno from 1 to 500000. Either a record is intable1 or table2. I want to find records (or SerialNo's) that are inneither table (if deleted by accident etc). What would be the sql query?I'm using SQL 6.5thx
Can any one tell me ..is there any restriction on the total number of records that can be added to a table ( SQL Server 2005) .Because .. i added 100 records to a table through my program..but i was able to c ..only the last 20 records which is being inserted..can any one tellme any way to increase the total number of records..
pls..help me..solution is required immediately.........
Can you help me to resolve my problem ? I have to do a simple daily backup system. Source : Flat File; Destination : SQL Server. I want to use the Slowly Changing Dimension component to backup only the new and updated row from my source (Flat File) and put them into SQL Server.
But how can I do to detect deleted rows from my source ?
Any suggestions ?
If it's not clear enough, please ask for more details !
I am having a hard time understanding triggers. My goal is to put a trigger on table x where records are UPDATED or DELETED. When this trigger fires I need to take the record ID and put the ID modified record into table y with the date modified. so basically logging the recordid changed with the getDate()
I don't quite understand how to get the rowid of the modified record.
We are looking to store a large amount of user data that will bechanged and accessed daily by a large number of people. We expectaround 6-8 million subscribers to our service with each record beingapproximately 2000-2500 bytes. The system needs to be running 24/7and therefore cannot be shut down. What is the best way to implementthis? We were thinking of setting up a cluster of servers to hold theinformation and another cluster to backup the information. Is thispractical?Also, what software is available out there that can distribute querycalls across different servers and to manage large amounts of queryrequests?Thank you in advance.Ben
We have an app that uses triggers for auditing. Is there a way to know the order that the records were inserted or deleted? Or maybe a clearer question is.... Can the trigger figure out if it was invoked for a transaction that "inserted and then deleted" a record versus "deleted and then inserted" a record? The order of these is important to our auding.
When i debug a trigger is it possible to add a WATCHon the INSERTED or DELETED?I think not, at least I couldn't figure out a way to do so.Does someone have a suggestion on how I can see the values?I did try to do something likeINSERT INTO TABLE1(NAME)SELECT NAME FROM INSERTEDbut this didn't work. When the trigger completed and Iwent to see the TABLE1, there were no records in it.Are there any documents, web links that describe waysof debugging the trigger's INSERTED and DELETED?Thank you
I guess this is a fairly common topic but couldn't find the right words to find anything in a search.
What I'm getting at, is there any tsql functions or combination of commands for the following.
You have identity columns in your tables, if you set the a seed and autoincrement, I enter in rows 1 -10 and then I delete 4, 6, 7, 8.
My next new record uses 11. Is there any logic that allows you to check and reuse 4, 6, 7 & 8 described above? Not looking for something that consists of having to create an extra ID table for each table and handle configuring what the next available number is everytime an Insert or delete is called.
I need to script out data in several tables (30+) and then reload those tables on a different database. The "target" database table data will get overwritten each time, not appended to. Several of these tables have numerous foreign keys and other constraints that must be dealt with. None of them have more than 100 records or so. Then I need to keep this script in source control and reference it as part of a post-deployment step with a Visual Studio DB project.
I have redgate, dbghost and visual studio. Are there any tools will generate the script for me complete with dropping all the constraints, truncating the tables, then re-adding the constraints back? Or, am I looking at simply scripting the table data out, then modifying the script myself to add in the steps to drop/readd the constraints? Further, since some of these tables might have relationships, not just to other tables, but to those in question, there will likely be an order in which the tables need to get loaded.
The approach I am thinking will be needed, since ive done this sort of thing before except with SSIS, is to run something like dbghost or the "generate scripts" option in SSMS as a starting point. From there, change the order around as needed, etc.
