Lots Of Columns To Check/convert - Best Practice?
Jun 27, 2007
I've got about 6 formats of flat files coming through data flows & heading into a relational db, then later to a data warehouse. In each file type, I've got about 60-70 columns to perform basically two levels of validations on - first is straight data type conversions, then 2nd is finer level stuff. Some of the data from each file type overlaps in other files, so for instance some lookup codes are maintained for both.
The data is pretty dirty, so I'm keeping everything as varchar coming into the staging area, just so I can get the data in the system, b/c the users demand on some form of of the data making it into the system, no matter the dirtiness.
So then I'm running my two steps - first converting data types from varchar to bit/datetime/int, etc. as applicable. And then , I'll be running finer levels of validation, doing range checks, etc. My question is - with so many columns, what's the most efficient & best way of doing all these checks in the data flow, and recording errors out of each check? Do I put one column check after another with the success constraint, do the lookup/range check/other, and then record the error if an error is encountered, then move on to the next column? Or would it be better to multicast out the stream to 60 flows and do all the needed checks, then union all the good stuff back together at the end? Anything to help save some headache - b/c this data is dirty and there's a lot of it.
View 4 Replies
ADVERTISEMENT
Feb 7, 2006
What's the best practice for adding / editing a record into a database with lots of fields ?I am not talking about the mechanics of it, as there are a lot of trivial examples using ADO.NET, stored procs, etc.
Deleting is easy, you just pass in (a few) primary key/keys to uniquely identify the record.
But in the real world when you have, say, a table with 100 fields! Do you code the INSERT sproc by hand, with 100 parameters... then call it with your ADO.NET code ? sounds like a lot of work to me...
What about updating! That's even worst, sometimes you may need to update only 3 or 4 fields, but using sprocs you would have to pass the whole 100 parameters in again, and "update" the whole record (when in fact you are only changing 3 or 4 fields).
With the update i could write different sprocs targeting only the fields i wish to update, but that sounds like duplicating work, vs having one generic update proc.
Sometimes i just feel like bypassing sprocs and having inline sql as it would be less work... but i know it is untidy.. and more potential to be buggy.
So come on guys (and gals)... let's hear your thoughts on how you would handle the insert / update scenarios when you have lots of fields ? Northwind examples are too trivial :-)
View 1 Replies
View Related
Aug 7, 2007
I'm sending a lot of columns through my derived column transform, checking for empty strings from a flat file - I was wondering is there a way that I could "script out" all the transforms instead of enduring this click hell that I'm stuck in inside the derived column transformation editor. I've got probably 100+ columns to configure with the following sort of transform....
Replace Col1
TRIM(Col1) == "" ? NULL(DT_WSTR,2) : Col1
Basically - if the string is empty, then throw Null in the data stream. I'm about a third the way through but it would really be nice if there was a quicker way. Even with the most efficient copying & pasting & keyboard shortcuts, it's still painful.
View 1 Replies
View Related
Apr 14, 2015
Content table
deleteddate revokeddate
1/1/2001 null
null 2/2/2003
4/5/2004 null
I am trying to create a stored procedure where.I need to loop through every row of the table and check if there exists a date in either of the column. If date exists in either of the column for every row, I need to update some other stuff.For the above table, it meets my requirement, so I need to update. However if the table is as below, it doesn't meet the requirement, so I don't need to update
deleteddate revokeddate
1/1/2001 null
null null
4/5/2004 null
I started off as below, But I am getting incorrect values.
Declare @deleterevoke bit
Declare @DelRevId int
Declare @DelRevnumrows int
Declare @tempDeletedDate datetime
Declare @tempRevokedDate datetime
[code]....
View 4 Replies
View Related
May 21, 2014
I keep thinking this can be done in one query:UPDATE H
SET Phone1 = NULL
FROM Dialer.dbo.Agentless_Hold H
INNER JOIN Dialer.dbo.PhoneLookup D ON H.Phone1 = D.Phone;
UPDATE H
SET Phone2 = NULL
[Code] ....
I need to NULL any phone number in the Hold table where the number is in the Lookup table.
View 5 Replies
View Related
Nov 25, 2014
I have two tables table1 and table2. I want to check a value from table1 against 4 different columns in table 2. What would be the most optimized way to do this. I came up with this SQL but it runs forever.
select * from table1 a
where
(a.id in (select orig_id from table2 where exctn_dt >= '01-OCT-14')) or
(a.acct_intrl_id in (select benef_id from table2 where exctn_dt >= '01-OCT-14')) or
(a.acct_intrl_id in (select send_id from table2 where exctn_dt >= '01-OCT-14')) or
(a.acct_intrl_id in (select rcv_id from table2 where exctn_dt >= '01-OCT-14'));
View 5 Replies
View Related
Apr 9, 2008
hi need help from the wonderful people here
i am working on project and convert a OLD DOS program to SQL server on the web
this is the resalt i need to show
napipot
Fname
new_unit
day1
day11
day111
day2
day22
day222
day3
day33
day333
day4
day44
day444
1
aaaa
1_1
A
C
Q
D
E
y
2
bbbb
2_2
-
E
A
d
3
cccc
3_3
B
F
B
z
f
4
dddd
1_1
-
*
5
eeee
4_4
C
A
C
--------------------------------------------------------add space row-------------------------
result row
Ok
OK
OK
ok
B
ok
ok
ok
ok
A
B
C
i need check only the
A
B
C
if not show the missing A B C
like this
A on the left
B on middleC on right
View 1 Replies
View Related
Jul 25, 2014
I have a case where if the Id field is a specific value, I don't want to allow null in another field, but if the Id value <> a specific value, null is ok.
In the example below, inserting the first record should succeed, the second should succeed, and the 3rd should fail. Right now the 2nd two fail. I gotta be missing something easy, but I can't figure it out.
USE tempdb
GO
IF OBJECT_ID('tempdb.dbo.CheckConstraintTest') IS NOT NULL
DROP TABLE tempdb.dbo.CheckConstraintTest;
CREATE TABLE CheckConstraintTest
[Code] .....
View 4 Replies
View Related
Mar 10, 2008
Hi,
I have an SSIS Package. I am using script component to loop through input columns and their values. I am not able to do Null checking. The code is as below. In place of dashes , I want to do null checking but am not able to do. I tried vbNull, IsNull, TypeOf, System.dbNull but nothing is working. I guess am missing something here. Can anyone help me with this.
For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
columnValue = rowType.GetProperty(column.Name)
If Not --------------- Then
RowData += ((FormatElement(column.Name) + ReplaceSpecialChars(columnValue.GetValue(Row, Nothing).ToString()) + FormatElement(column.Name, True)))
End If
Next
Thanks in Advance.
View 11 Replies
View Related
Mar 5, 2008
Hi
How can I check whether an nchar column contains Arabic character set? The only allowed values that can go into this column are English and Arabic.
Thanks,
Salman Shehbaz.
View 4 Replies
View Related
Jul 11, 2006
I'm using a stored procedure to add fields to an existing table.
These fields must have check constraints and I need to use one T-SQL batch.
In Sql2000 Ok. In Sql2005, if table exists, I get error "invalid column xxxxx" in Add Constraint statement before new column creation.
the code is
Declare @Setup bit
Set @Setup = 1
if @Setup = 1 Begin
--Alter Table MyTable Add MyField Numeric(1, 0) Not Null Default 3
Exec mySp_Add_Column 'MyTable', 'MyField', 'Numeric (1, 0) Not Null', '3'
If IsNull(ObjectProperty(Object_Id('xCK_MyTable_MyField'), 'IsConstraint'), 0) = 0
Alter Table MyTable Add Constraint xCK_MyTable_MyField Check (MyField >= 1 And MyField <= 3)
End Else Begin
-- drop column
End
GO
If MyTable does not exist and, naturally, I add it before of check constraints (using another Sp which add tables) ok.
If I add FK to new fields, ok.
Now I have to split batch in two parts as workaround...
Can anyone tell me if this is a bug or a "fix" for previous versions?
Many thanks,
Giulio
View 7 Replies
View Related
Aug 30, 2004
Hi,
I have some data in one table.
I need to get the data in the same table, into different rows.
I mean to say that, I have data like
InvNo InvAmt GstAmt
12 1900 79
and I need to get data like
InvNo Amt
12 1900
12 79
Can anyone help me as to how can i get data in this format.
Thanks.
View 2 Replies
View Related
Jun 25, 2015
I am looking for the best way to check to see if any columns are still NULL in a record. I have a form that gets filled out by users and the values entered into TableA. There are 6 columns in the table, 5 are responses and column6 indicates if the record is complete. So I want a way to see if all of the first 5 columns are NOT NULL and, if so, mark column6 with a 1.
I am thinking this would be a good thing for a trigger on INSERT or UPDATE to check to see if the first 5 columns are filled in and then mark the record as complete.
View 5 Replies
View Related
Jul 11, 2001
My table:
Table1
T1T2T3T4T5
---------------
abcde
How to query to have following result:
Table2
abcde
---------------
T1T2T3T4T5
T1T2T3T4T5
T1T2T3T4T5
T1T2T3T4T5
T1T2T3T4T5
Thanks in advance.
Janasha H
View 1 Replies
View Related
Aug 21, 2014
I've a table like below
ColA ColB
A12
B22
C45
A56
A34
B23
and I need to convert the rows into column names like below
ABCA1A2B1
122245563423
View 1 Replies
View Related
May 29, 2007
how to convert rows into columns.
Data in table
empcode basic
1 1000
2 2000
Required output
empcode 1 2
basic 1000 2000
View 3 Replies
View Related
Nov 13, 2007
Hi there
Assuming I have the following data where the header represents budget cost and the value represent no. of projects
<2K 2K-5K >5K
--------------------
10 15 5
For the above table: the following is my sql:
SELECT
SUM (CASE WHEN PRJ.BDGT_CST_TOTAL<2000000 THEN 1 ELSE 0 END) <2K,
SUM (CASE WHEN PRJ.BDGT_CST_TOTAL>=2000000 AND PRJ.BDGT_CST_TOTAL<5000000 THEN 1 ELSE 0 END) 2K-5K,
SUM (CASE WHEN PRJ.BDGT_CST_TOTAL>=5000000 THEN 1 ElSE 0 END) >5K
FROM
PRJ_PROJECTS AS PRJ
How do I program it in sql such that the data will be displayed as below? Thanks
Budget_Range No_of_prj
-----------------------
<2K 10
2K-5K 15
>5K 5
View 5 Replies
View Related
Jul 23, 2005
Hi All,I need to help with converting rows to columns in SQL2k.Input:Id Name Role58Ron Doe Associate58Mark BonasDoctor59Mike JohnsonDoctor59John SmithAssociate102Chris CarterAssociate102Ron Doe Associate102James JonesAssociateOutput should look like:IdDoctorAssoc1Assoc2Assoc358Mark BonasRon Doe NULLNULL59Mike JohnsonJohn SmithNULLNULL102NULLChris CarterRon Doe James JonesThere could be more than 3 associates in the input but I only need 3above columns for associates.I used following query:SELECT Q.sales_id,doctor2= (SELECT Q2.name FROM view1 Q2 where Q2.role = 'doctor'and Q2.sales_id = Q.sales_id),assoc1= (SELECT Q2.name FROM view1 Q2 where Q2.role ='associate' and Q2.sales_id = Q.sales_id),assoc2= (SELECT Q2.name FROM view1 Q2 where Q2.role ='associate' and Q2.sales_id = Q.sales_id),assoc3= (SELECT Q2.name FROM view1 Q2 where Q2.role ='associate' and Q2.sales_id = Q.sales_id)FROM view1 QGROUP BY sales_idand I get this error "Subquery returned more than 1 value" since thereare multiple associate for Id 102.Thenks
View 3 Replies
View Related
Apr 24, 2006
hi
i have a sample data like
patientno visit experimentdate
101 1 23-Dec-2004
101 2 23-Mar-2005
101 3 23-Jul-2005
102 1 23-Dec-2004
102 2 23-Mar-2005
102 3 23-Jul-2005
i want it to display like this
patientno visit1 visit1date visit2 visit2date visit3 visit3date
101 1 23-dec-2004 2 23-Mar-2005 3 23-Jul-2005
102 1 23-dec-2004 2 23-Mar-2005 3 23-Jul-2005
..please suggest how can we do it
View 4 Replies
View Related
Feb 28, 2008
Hi I have a stored proc that pulls in data, I have four columns that, where there is data I need to times by 100 but when empty to remain empty, currently where empty I end up with 0's
Do i need to do a convert or something? can someone help me out thanks
Code Snippet
USE [WINDRUSHDAL]
GO
/****** Object: StoredProcedure [dbo].[spgiffordstelesalesdev2] Script Date: 02/28/2008 12:34:21 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author: <Author,,Name>
-- Create date: <Create Date,,>
-- Description: <Description,,>
-- =============================================
ALTER PROCEDURE [dbo].[spgiffordstelesalesdev2]
AS
BEGIN
SET NOCOUNT ON;
SELECT
BLCUSA AS Company,
BLCUSB AS Account,
BLDELN AS DeliverTo,
blcnam AS [Account Name],
BLCTS1 * 100 AS Monday,
BLCTS2 * 100 AS Tuesday,
BLCTS3 * 100 AS Wednesday,
BLCTS4 * 100 AS Thursday,
BLCTS5 * 100 AS Friday
FROM
LIVEAS400.S65C422B.WRFDTA.PARDADR AS PARDADR_1
WHERE
(BLCUSA = '2')
Union
SELECT
abCUSA AS Company,
abCUSB AS Account,
0 AS DeliverTo,
abcnam AS [Account Name],
abCTS1 * 100 AS Monday,
abCTS2 * 100 AS Tuesday,
abCTS3 * 100 AS Wednesday,
abCTS4 * 100 AS Thursday,
abCTS5 * 100 AS Friday
FROM
LIVEAS400.S65C422B.WRFDTA.PARNADR AS PARNADR_1
WHERE
(abCUSA = '2')
END
View 7 Replies
View Related
Apr 10, 2008
Hi,
I have a table which consists of some rows
i want to get rows as columns
For eg:
table1
id name dept
1 x a
2 y b
3 z c
i want to write a select query to fetch result as
1 2 3
x y z
a b c
can anyone help please
View 10 Replies
View Related
Sep 15, 2002
I need to find column names if any after 2 “check” columns.
Scenario: I have a database, with approx 400-1500 tables, depending on installation of software. The software is structured so that, when it synchronizes the SQL database it will create all the columns e.g. custacc, custname etc. and then it will always put in two check columns “CheckOne” and “CheckTwo” these two columns has to be the two last ones. In 99.9 this always works fine, but sometime if the users creates a new field in the software, when it synchronizes the new field “lands” behind the two checkfields, which is not good.
So what I am after is a script, which can run through all user tables, tell me if there are columns after the two checkfields and list those tables if any.
Any help would be greatly appreciated.
Cheers
Henrik.
View 3 Replies
View Related
Apr 13, 2007
Hi Guys
I need your help again, I am try to update several columns and the data type is 'money'.
Below is the code I have used:
UPDATE CAT_Products SET
UnitCost ='10.00',UnitCost2 = '10.00',UnitCost3 = '10.00',UnitCost4 = '10.00',UnitCost5 = '10.00',UnitCost6 = '10.00'
WHERE ProductCode = '0008'
But it will not update, instead I get this error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>[Error] Script lines: 1-9 -------------------------- Disallowed implicit conversion from data type varchar to data type money, table 'dbo.CAT_Products', column 'UnitCost'. Use the CONVERT function to run this query.
More exceptions ... Disallowed implicit conversion from data type varchar to data type money, table '.dbo.CAT_Products', column 'UnitCost2'. Use the CONVERT function to run this query.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The error message indicates that I need to use the convert function. But the columns data type is set at 'money' not 'varcher' . So do I need to convert data type to 'varcher' in order to update and convert back to data type 'money' when update complete? Or do I need to indicate in the update statement that data type is already 'money'? I am not sure how I would either.
Thanks
View 2 Replies
View Related
Nov 11, 2011
I have a table that imported as varchar. Most of these columns need to be in a numerical format. How can I convert a table with columns named column0 (needs to be int),column1 (stays varchar), column2(needs to be int), and column 3(needs to be int)?
View 4 Replies
View Related
Sep 17, 2014
I work with SQLite and need to write a query the old school way to convert rows to columns. If it was MS SQL I would use pivot to get the expected result. However this is SQLite I cannot use pivot.
Sample data:
create table t1 (id int, Dept char (1), Total int);
insert t1
select 1, 'A', 100
union
select 2, 'B', 120
union
select 3, 'C', 140
union
select 4, 'D', 150;
How do I use LEFT OUTER JOIN to produce result similar to the below?
SELECT 'Total' AS Dept,
[A], [B], [C], [D]
from
[code]....
View 7 Replies
View Related
Apr 3, 2008
Hello All,
I have one table with 9 different columns for e-mail address like mail_address1, mail_address2 etc....
now i want to insert serial no 1 for mail_address1, serial no 2 for mail_address2 like wise for a particular id.
Current Format :-
code mail_address1 mail_address2 mail_address3
1111 abc@yahoo.com xyz@yahoo.com null
Required format:-
code sr_no mail_address
1111 1 abc@yahoo.com
1111 2 xyz@yahoo.com
1111 3 null
i hope that i am clear with my question.
Can any one help me?
Thanks
Prashant
View 4 Replies
View Related
Apr 4, 2008
Hello,
Table - 1 Format:-
Line_No Entity_Id Mid
1157530
2157540
3157550
4157560
5157570
6157580
7157590
81575100
Now i need the data into below Format :-
Line_NoEntity_Id Ticker Mid1Mid2 Mid3 Mid4 Mid5 Mid6 Mid7 Mid8
11575 COFP 30 40 50 60 70 80 90 100
Can any one help me out?
Thanks
Prashant Hirani
View 1 Replies
View Related
Nov 14, 2014
I am converting some access to SQL at work and I am having a problem with a query that uses the access FIRST function
My query looks like this in access
SELECT
TBL1.ID
, FIRST(TBL1.Code) AS FirstCode
FROM
TBL1
WHERE
TBL1.Marker = 'X'
AND Left(ID,1) IN('A','B','C','D','E')
GROUP BY
ID
I have tried to convert it using a slightly different method. So instead of picking the first value of the Code field that appears in the select, (which is what access would do and SQL can’t do as far as I’m aware). I first select the ID and code with the highest value associated with them. The below script is what I did. I would’ve thought they would produce the same number of records but they don’t.
SELECT
ID
, Code FirstCode
FROM
(
SELECT
ID
, Code
, MAX(Value) MaxValue
FROM
TBL1
WHERE
TBL1.Marker = 'X'
AND LEFT(ID,1) IN('A','B','C','D','E')
GROUP BY
ID
, Code
) T1
View 3 Replies
View Related
Jan 13, 2008
Hi,
I have a table with 5 columns, from that I need only 3 columns , but the columns should be rows in the output
Ex Table 1
col1 col2 col3
Now I want col1 as one row n col2 as 2nd row and col3 as 3rd row
What query should I write to get this?
Will union work
Thank You
View 1 Replies
View Related
Oct 8, 2015
How can I convert the following data
Create Table tbl_Customer_Bank_Details
(
[CustomerID] int,
[CustomerName] varchar(50),
[AccountType] varchar(50),
[BankAccountNo] varchar(50)
)
[Code] ...
into the following format
Create Table tbl_Customer_Bank_Details2
(
[CustomerID] int,
[CustomerName] varchar(50),
[AccountType] varchar(50),
[BankAccountNo1] varchar(50),
[Code] ....
I am looking at this [URL]
how to do this in SQL 2000.
View 16 Replies
View Related
Apr 24, 2015
i have a table with dob and test results , i am trying to pull the data from the table and converting rows columns , below is the table i am using . i used to pivot to do this .
create table #TEST_RESULTS
(ID INT,NAME VARCHAR(10),DOB DATETIME,DAYS_SINCE_BIRTH_TO_TEST INT,TEST_RESULTS INT )
INSERT INTO #TEST_RESULTS
VALUES(1,'A','2015-01-01' , 0 ,1)
,(1,'A','2015-01-01' , 0 ,1)
,(1,'A','2015-01-01' , 1 ,3)
,(1,'A','2015-01-01' , 2 ,6)
[code]...
View 9 Replies
View Related
Apr 30, 2008
Hi I have a 'settings' table similar to:
* Setting_ID
* SettingGroup
* SettingSubGroup
* SettingKey
* SettingValue
example data:
1, Comms, Com1, Speed, 9600
2, Comms, Com1, Parity, N
3, Comms, Com1, DataBits, 8
4, Comms, Com1, StopBits, 1
5, Comms, Com2, Speed, 2400
6, Comms, Com2, Parity, E
7, Comms, Com2, DataBits, 7
8, Comms, Com2, StopBits, 2
I want to have a query like
SELECT SettingKey, SettingValue FROM Settings WHERE SettingGroup='Comms' GROUP BY SettingSubGroup
where the result set has a single row for each distinct value of SettingGroup. I want the result set to look like:
SettingSubGroup, Speed, Parity, DataBits, StopBits <--column headings
----------------
Com1, 9600, N, 8, 1
Com2, 2400, E, 7, 2
I've had a look at the PIVOT command but it seems to require an aggregate function... is there a way to simply flip from rows to columns?
View 1 Replies
View Related
Apr 10, 2014
I need find out the number of columns in flat file before i process that particular file.
I have file name in @filename variable and file path is @filepath variable.
But do not not that how i will check the column name in before i will process that file.
@filePath = C:DatabaseSourceFilesCAHCVSSourceFiles
And I am using for each loop container to read the file one by one and put the file name in @filename variable.
and my file name like
Product_20120607060930.txt
Product_20130708060930.txt
My file structure is:
ID,Name,City,Country,Phone
1,Riya,Pune,India,454564
2,Jiya,New Jersey,India,454564
3,Riya,St Louis,USA,454564
4,Riya,Belleville,USA,454564
5,Riya,Miami,USA,454564
Now what i have to do is i need to make sure that ID,Name,City,County,Phone is there in flat file. if it is not there then i have to send mail to client saying that file is not valid.
Let me know how i will do it.I need to also calculate the size of flat file.
View 0 Replies
View Related
