SQL Server Data Validation
Mar 18, 2008
Hi.
I have one question regarding SQL Server replication.
I have setup SQLServer merge replication at my work, one of our subscriptions has unreliable connection to the publisher, after setting up the replication to this subscriber, it happened once that the connection was really bad so the merge agent for this subscription was failing before synchronizing data changes, and after that , the connection went good & the merge agent completed the synchronization successfully.
After a while I decided to validate all subscriptions, I discovered that this subscription has different row count for many articles although the merge agent was finishing all synchronizations successfully!!
so my question is:
how could this happen? I mean finishing the sync with success and still have different records!!
I thought that SQL Server will keep make the sync fail just because it can't do it!!
Is it right or there is something I misunderstood here?
How can I prevent this from happening again? at least I want SQL Server to inform me by failing the merge agent synchronizations.
View 14 Replies
ADVERTISEMENT
Sep 17, 2015
I want to do something with error checking in my company. For this we have a selection of different tables and the data needs to meet various validation rules else it is classed as an error.
To deal with this I'm currently thinking of this approach:
1. Create a view pulling all of the various data together from the multiple tables.
2. Create an empty 'errors' data table.
3. Create an Excel file with a button to call a Check for Errors Script
Then in the the script:
1. Clear the 'errors' data table
2. Call multiple scripts, each of which uses the new view, applies the checks for that specific error and writes any erroring data into the 'errors' data table (along with a text string with the unique error code for filtering / sorting purposes).
3. After calling all the scripts, the table can be refreshed in excel when when used with a pivot table can show the various errors, and let us drill down into all the data so we can fix them.
Also.. Ideally, I'd like some way to write comments in an excel column for each entry and error code and be able to write that back into a comment table.
View 5 Replies
View Related
Jun 25, 2015
In C# .NET I have the possible to create some validations of my data, with regulary expressions. Do SQL have the same feature? I will like to do an data validation of all my insert statement inside the sql-server. Is that possible?
View 5 Replies
View Related
Apr 27, 2008
I'm developing a database-driven program using SQL server 2000 and Visual Basic 2005.
Most of the guys say professional programming is doing the validation stuff (such as the constraints and data integrity stuff like" [0-9][1] " and the use of LIKE IN keywords etc.) in the databse itself.
say i did the data validation contraints in SQL server itself. and now i connect the database with the interface made in 2005. and say a person enters some invalid data through the interface. but the error messages are generated by SQL server. how am i to display the SQL server generated error messages in the VB made interface??
PLS HELP ME .. if the question is not clear pls tell so that i can explain it further.
PLS PLS PLS HELP ME
View 1 Replies
View Related
Apr 27, 2008
I'm developing a database-driven program using SQL server 2000 and Visual Basic 2005.
Most of the guys say professional programming is doing the validation stuff (such as the constraints and data integrity stuff like" [0-9][1] " and the use of LIKE IN keywords etc.) in the databse itself.
say i did the data validation contraints in SQL server itself. and now i connect the database with the interface made in 2005. and say a person enters some invalid data through the interface. but the error messages are generated by SQL server. how am i to display the SQL server generated error messages in the VB made interface??
PLS HELP ME .. if the question is not clear pls tell so that i can explain it further.
PLS PLS PLS HELP ME
View 4 Replies
View Related
Jun 24, 2014
I am currently in the process of migrating data from Sybase to Sql server and would like to know how to test the data migrated.
As of now, we took one table data from both source and destination and compared it in Excel to check if the data migrated looks good (note, we used SSIS to migrate data). However, I would like to check if there are any other best & easy ways to apprach data validation post migration.
View 3 Replies
View Related
Jun 10, 2015
I Have a table with #Sample like below
=================================
#Sample
id int,
SSN varchar(20),
State varchar(2)
Â
Sample Data:
ID SSN STATE
1 999-000-000 AB
2 979-000-000Â BC
3 995-000-000Â CD
=================================
We used filter logic based on the SSN & State.
We are passing these values through variables like
Declare @State varchar(2)
Declare @SSN varchar(20)
While run time these values are lets suppose @SSN = '999-000-000' & @State='ABC'
Now the Result is displayed with the state data Like 'AB' only.
Output: 1 999-000-000 AB
instead it should give system generated error.
Here I have 2 Questions:
1. Why it is taking 1st 2 Charecters?
2. Why it does not have any system generated for length?
I can do validation with Length function for these 2 variables however if have 100 variables then it should not feasible case. So, what is the reason behind?Â
View 5 Replies
View Related
Jun 15, 2007
Could I ask how to spit the data into training and validation sets when doing data mining?
Thanks
View 1 Replies
View Related
Sep 22, 2007
Hi guys I am fairly new to t-sql. I am sure there are SPs or scripts that I can use to create a procedure that will do the data validation in the staging table...
Client send us data and often some of the records have bad values... what i have to create is a process that will check for those values and update a flag in the staging table for each column if the data is not valid....
Please help me out if you have something that can be used for this.
Thanks a lot help in advance.
View 5 Replies
View Related
Jun 13, 2006
Being relatively new to SSIS, I'm looking for advice, or a best practice, regarding data validation before extracting the data for a transformation.
One of my project's require that certain data be validated in staging tables before it is loaded. The validations include checking for null values, verifying that a field is populated with apropriate values etc... The entire batch of data (good records and bad records) may be rejected depending on the validations.
I have a couple of different thoughts on how this could be handled...
Run a series of validation queries on the data before executing an SSIS package
Run some kind of validation transformation (does one exist or should I write a custom transformation?)
Place contraints on the target tables so that bad records error out on the load
Something else... I could be missing the completely obvious
#3 doesn't seem to viable as the entire load may be rejected if some of the data is bad...
Any thoughts?
View 6 Replies
View Related
Nov 28, 2007
Hello all...I am trying to validate that the new work date that is being imported from the text file does not exist in the table. So in other words I do not want duplicate data. This is what I haveSqlDataReader dr = new SqlDataReader(); SqlParameter sp = new SqlParameter("@WorkDate", Data.SqlDbType.datetime, 8, Data.ParameterDirection.Input); if (dr.HasRows) sp = "@WorkDate"; else *How does that look? Am I in the ball part…
View 5 Replies
View Related
Jan 31, 2008
I new here so hopefully I'm asking this question in the correct forum. I'm have a flat file that contains numbers that I need to verify that they begin with certain prefixes so they load to the correct client. For example in the flat file if I'm loading data for client A and their account numbers begin with 045XXXXXXX then it loads the data. But if there is a record that begins with 037XXXXXXX it should be loaded to client B instead and that records gets written to a error file.
So to summarize what I need , I'm looking for a check to kick out records if I'm loading client A's data versus if I'm loading client B's data.
Can this be done in a Conditional Split ?
Shanon
View 9 Replies
View Related
Oct 18, 2007
Hello everyone,
I have spent hours trying to figure out the validation issue with SQL Server Reporting Services with no luck.I am hoping someone can help me with the issue.
Basically I am looking for a way to validate the data entered by the user in the parameterized Reports.
say, the field is defined as int datatype. If user enter String, it should display the message saying string cannot be entered. Currently, My report does not check the user entered datatype, runs the underlying query and displays the error thrown by the DB ( oracle in my case).
I get :
An error has occured during report processing
Cannot read the next data row....
ORA-01722: invalid number
Instead of this, validation even before query is executed would have been perfect. Is there a way to do that?
Thanks in advance for the help
Phewa Taal
View 3 Replies
View Related
Aug 12, 2007
Hi all,
I want to make a conditional split based on the data type provided by the input.
For example : If the comming (Column x) is of data type (numeric) then pass , else do not pass.
(pass = Case 1
Do not pass = Case 2).
Is there any way for doing so ?
View 3 Replies
View Related
Apr 4, 2007
I am working on a coupon redemption project that appears to require dynamic data validation and need some help. For example, a coupon could require a person to buy XXX number of XXX items from XXX manufacture before the transaction is approved. The logical operator between each validation could also be “AND� and “OR�. My first attempt at this has been something along the lines of creating a coupon table with a child table of validation logic. However, being able to apply this logic (and at what layer) has turned out to be a challenge. I am very concerned about speed as all requirements have to be met before a transaction is approved. Any help with trying to figure out the best solution for this would be great@!
Coupons
C_ID
C_DESC
MFG_ID
MFG_OFFERNUM
Coupon_Validation
CV_ID
C_ID
MFG_ID (Manufacture)
MFG_PARTNUM (Manufacture Part Number)
CV_QTYREQUIRED (Number to meet requirements)
CV_ITEMLOGIC (AND, OR)
View 1 Replies
View Related
Nov 27, 2007
I need some suggestions on validating a string of text based on some business rules using T-SQL. I have a string similar to the following:
This is some text (1.00); this is some more (there's more text here) text (2.00); this is yet more text (1.10)
The above example illustrates a valid string. You'll notice the multiple sets of parens The parens that contain numbers must be followed by a semi-colon except for the last set of parens. Furthermore, any parens followed by a semi-colon must contain only numbers and not text. I can easily identify the positions in the string by using PATINDEX.
Get the closing paren not followed by a semi-colon, ignoring the last closing paren PATINDEX('%)[^;]', @string)
Get the closing paren followed by a semi-colon, ignoring the last closing paren PATINDEX('%);%', @string)
My question is, is there a way to quickly validate the data between the parens without visiting each character. This is SQL 2000 so using CLR regular expressions is out.
Thanks.
View 1 Replies
View Related
Feb 15, 2013
We have the following scenario: We receive CSV files every month for which SSIS packages were built to process the data. The following problems occur from time to time:
1. The structure of the CSV file changed (e.g. column added or removed)
2. There were no footers in the data, but now footers started to appear
3. Date format changed (e.g. used to be mm/dd/yyyy, but became mm.dd.yyyy)
4. Number format changed (e.g. from 2000 to 2,000)
Currently we have person who manually opens each file, and using our "validation document" validates to ensure none of these or similar problems occur. We would like to move away from this manual process if possible. I understand that items 3. and 4. could be caught by loading data into a staging table with VARCHAR data types, and performing validation before moving it any further.
Item 2 is a bit questionable (meaning depending on the footer size SSIS load could fail or not).
Item 1, however, is a sure fail of the SSIS package that directly loads the data into a table.
Thus I feel the two possible options are:
1. Create a custom script that will run through the file, row by row, apply all the necessary validations and report an error or continue if all checks out
2. Use some 3rd party tool to validate the files (semi-manually) before kicking off the SSIS processing.
View 3 Replies
View Related
Oct 5, 2015
Im newto SSIS. I want to develop package for data validation.
FirstName
1. Mandatory  field checking: if Null, reject the record
 2. If field length > 50, then reject the record
SSN
1. Â If field length > 12, then reject the record
2. If SSN is not in valid format, issue warning and process rhe record  without SSN value.
3. Valid format: 9 digit numeric values should present after striping off  all non-numeric characters.
4. Only send 9 digits to MDM
Like these i have 30 rules. And I have to shop the error msg if the validation fails like "Mandatory feild is missing".
View 2 Replies
View Related
Aug 29, 2007
Hi,
I wanted to know more about validation of SSRS parameters. I have a simple report which has a parameter called startdate of DateTime datatype. The datetime parameter in SSRS takes manual input as well. So, the user can enter any junk value. I want to ensure that the input parameter is in correct format and I want to display an error msg when the format is incorrect. My report has the following VB code for validation:
Public Function Validate( ByVal startdate As String) As Boolean
If IsDate(startdate) = True Then
Return True
Else
Return False
End If
End Function
And my report has a textbox which has the expression property set to;
=Code.Validate(Parameters!startdate.Value)
the textbox on the report has to display if the entered date is valid or not.
But, when i enter an erroneous date, SSRS doesn't render the report and throws a generic error. This happens even before the code written for validating the parameter executes.
Also couldn't find a way to disable the manual input for the datetime parameter. Even that would solve the problem.
Another alternative was to make the startdate parameter as string, but i want the calendar control button to be provided for the user.
View 10 Replies
View Related
Jul 31, 2015
In my database some of the store procedures getting the data from xml nodes.so I need to implement the validation to xml data for prevent sql injection.
View 0 Replies
View Related
Nov 8, 2007
Dear all,
I am trying to executed a packege so that it loads data from from the excel file to the SQL Server Server database.
When I execute it, it prompts the following error message and 1 warning
The excel file has three colums, Week, Item and Value
Error 4 Validation error. Data Flow Task: OLE DB Source [94]: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Oracle" Hresult: 0x80040E37 Description: "ORA-00942: table or view does not exist ". Test - GET NW PERF 1.dtsx 0 0
Warning
Warning 1 Validation warning. Data Flow Task: OLE DB Destination [36]: The external metadata column collection is out of synchronization with the data source columns. The column "DAY" needs to be added to the external metadata column collection. The column "TCH_AVAIL" needs to be added to the external metadata column collection. The column "PDROP" needs to be added to the external metadata column collection. The column "P_HR" needs to be added to the external metadata column collection. The column "SFAIL" needs to be added to the external metadata column collection. The "external metadata column "VALUE" (90)" needs to be removed from the external metadata column collection. The "external metadata column "ITEM" (89)" needs to be removed from the external metadata column collection. Not in use - GET NW STATS.dtsx 0 0
Could someone give me a hand here.
Regards,
Ronald
View 1 Replies
View Related
Jun 6, 2015
I've created a merge replication a few days ago. it works correctly, but today when click synchronize publications, some of then encounter this error :
Data validation failed for one or more articles. When troubleshooting, check the output log files for any errors that may be preventing data from being synchronized properly. Note that when error compensation or delete tracking functionalities are disabled for an article, non-convergence can occur. (Source: MSSQL_REPL, Error number:
[Code] ....
View 2 Replies
View Related
Jul 25, 2007
I am using SSIS to populate a star schema.
The issue is in the data flow for loading and setting the Fact table dimension keys (the dimensions are all loaded fine). After 16 rather pedestrian Lookup Transformations, I have an escalating problem adding additional Lookup transforms to the Data Flow. The problem is not in execution; the problem is adding more transforms in design mode.
Lookup # Fields in Data Flow Time to validate that lookup
<17 47 Sub-second
17 48 2 sec
18 49 4 sec
19 50 8 sec
20 51 16 sec
21 52 32 sec
22 53 64 sec
While I€™m intrigued by the mathematical progression that is forming here, the issue is that I have at least 6 more Lookups to perform. I hope you can see my dilemma.
I have gone to where it takes a little over 4 minutes each to validate the lookup transform and its associated Derived Column transform and Union transform (Total 12 Minutes). Not only does this add up to many idle minutes to each design step, BUT it breaks the debugger as it pre-validates the ENTIRE data flow before it ever switches into debugging mode.
Some notes:
1. It doesn€™t matter what order the Lookup transforms occur in, the timings are exactly the same.
2. I tried many Data Flow execution optimizations, but they don€™t improve the validation times (or even get a chance to improve the execution times!)
I realize this may be somewhat of a unique problem.
Thanks for any help you are able to lend.
-Dave
View 3 Replies
View Related
Apr 17, 2007
I have a package set up basically with two consecutive data flows. The first flow takes data from an OLE DB Source and stores it into a Flat File Destination. The second flow uses this same flat file as a source, alters the data, and stores the data in the same flat file, overwriting the old file. I set DelayValidation to True on the flat file. Still, here are the error messages I am receiving:
Error: 0xC020200E at DO, Flat File Destination [7676]: Cannot open the datafile "C:Temp.txt".
Error: 0xC004701A at DO, DTS.Pipeline: component "Flat File Destination" (7676) failed the pre-execute phase and returned error code 0xC020200E.
I am new to SSIS, so I'm sure I have a setting wrong or something. Is the problem that SSIS is trying to write to a file from which it is simultaneously reading data?
Thank you.
View 6 Replies
View Related
Jun 15, 2007
Also when using cross validation, which descision tree should we choose?
Thanks very much
View 5 Replies
View Related
Jul 10, 2006
Hey all,
Quick question regarding validation!
I have a table of contact details containing the usual name, adresss etc fields.
I want to validate the fields Country and telephone together so that fopr example if country = 'UNITED KINGDOM' the telephone has to begin with +44 if it doesnt i want it to add the +44!
I can do this through writing a little program but just wanted to explore the possibility of doing this with SQL or functions already available with MSSQL Server
cheers
ed
View 3 Replies
View Related
Jun 7, 2006
Hi There,
I have a software where the frontend is created in VB and backend is created in SQL server. Our HO sends a file, this file has only one month validity. It has 2 files one is parent and one is child file. while uploading the file using software it creates the following files
Clipboard Clip File, Cursor File, DBE file, EXM, GLB file, MOD File, SQL server subscription file, MTB file, PAT file, PSK file, PTD file, QCM file, QTY file, SEM file, SMO file,STR file, TOP file and a VLP file.This files uploads questions and answers in database. and in SQL Profiler it runs the following queries
select dbname from master ..syslogins where loginname =UPPER ( 'S' )
select convert ( varchar ( 10 ) , getdate ( ) , 105 ) from sysusers where uid =0
select convert ( varchar ( 10 ) , getdate ( ) , 105 ) + ' ' + convert ( varchar ( 8 ) , getdate ( ) , 108 ) from sysusers where uid =0
select convert ( varchar ( 10 ) , getdate ( ) , 120 ) from sysusers where uid =0
All of them returns date. But after this it throws error file expired. I dont understand from where it checks date. and which file contains questions and answers
Can anyone please tell me which file might contain date validation and how do i remove it. I can mail the files for your kind reference.
Thanking you and waiting for some kind replies
mystical
View 1 Replies
View Related
Nov 18, 2005
HiI'm rather new to ASP.NET (was working with win forms before) and having a few problems with my first app. To the point - this is an internal system - every week about 50 branches send financial reports to the headquaters where those excel files are validated against data in sql server. after file is uploaded to the server I'm opening it using Excel object model and iterating through it line by line. Apart from format and datatype checking, for each line there are ~2 stored procedures invokedEverything is working actually but my concern is performance.. After about 800 rows it is getting really slow.. Appreciate any comments on this.thanksAM
View 1 Replies
View Related
Nov 28, 2007
Hi, can someone please explain some more how to use cross-validation in SQL Server 2008 (CTP)? I read here that it "is under Accuracy Charts in Business Intelligence Development Studio, in addition to being accessible programmatically via a stored procedure call". What is the stored procedure and input parameters?
I'm actually interested in doing so inside an Integration Services package, in a data flow task.
Thanks,
Gustavo Frederico
View 8 Replies
View Related
Mar 29, 2008
Is there some way to set up management studio to where a specific saved script (.sql file) won't run unless it's on a server in a certain domain (or at least on a specific server)? I know can store @@servername in a variable and then run the script inside of an IF block only if it's a certain server name, but I'm looking for something a little more bulletproof. I'll probably end up putting the script in a .NET app where I can control the connection string, but I'm wondering if there's a T-SQL alternative to this.
Thanks,
-Dave
View 5 Replies
View Related
Dec 13, 2007
Hi! I am using the Cross Validation tab in BIDS. Can you explain why my Liklihood Log Score is a negative number (according to BOL - meaning it is worse than a random guess) when my lift chart shoes that the alogorithm is significantly better than a random guess.
Also, the BOL definition for truenegative and falsenegative are hard for me to interpret. If I have a target state of "Yes", can you please put those definitions in yes/no terms for me?
Thanks.
Ann
View 3 Replies
View Related
Mar 28, 2007
Dear forum users,
I am a newbie in using MS SQL server with analysis services.
There seems to be no 'cross-validation' tool in MS SQL
which is frequently used in data mining and even statistics.
Is there anyone having similar difficulties?
Is there any solution like a small scripts to divide
the given dataset with multiple folds?
Your valuable comments and feedbacks would be appreciated.
Minnetongka
View 4 Replies
View Related
Dec 15, 2014
I have an SSIS package authored in SSDT for VS 2013 that cancels itself immediately after validation completes and execution commences. This behavior occurs when executed either in VS 2013 or from within SQL Server. No error messages are thrown in either the debug window or the log output (log is capturing everything). The only thing that occurs differently on this package as compared to another package I am able to execute successfully is that a command line window briefly flashes when the package cancels itself—but it is gone so fast I cannot read it. The last several lines of the debug output are as follows:
-----------------
Information: 0x40043006 at Merge Info, SSIS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Merge Info, SSIS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Merge Info, All Users CSV [2]: The processing of file "C:...AllUsers.csv" has started.
Information: 0x400490F4 at Merge Info, Lookup Org [47]: Lookup Org has cached 957 rows.
Information: 0x400490F5 at Merge Info, Lookup Org [47]: Lookup Org has cached a total of 26719 rows.
[Code] ....
What circumstances an SSIS package would cancel itself without throwing any errors?
View 2 Replies
View Related
