Flexible Back-end Data Handling In .net + Sql Project

Nov 23, 2005

Hi guys,

Got a problem now :( please help...

now we got a project handling records saved in a table in a sql
2000(will upgraded to 2005 soon) server. every month around a million
records will be inserted.

now user raised a request, that is, once criterios are matched, the
project should do some backend handle, for example, if

SELECT colID, fieldC, fieldD FROM dataTable WHERE fieldA =
'fieldA_valueB'

returns some recordset, for each @fieldC, @fieldD we shall do some
back-end trick, maybe

UPDATE dataTable SET fieldE = 'fieldE_valueF' WHERE fieldC = @fieldC
AND fieldD = @fieldD

let's say such a rule is named as rule01. Hope I'm expressing the
problem clearly?

my questoin is, shall we do this in Database side, using triggers, or
by informing our .net project to do it?

1. since the records are coming around 1 millison per month, how can we
handling the performance issue?

2. now the rules are still somewhat simple, seems at least we could do
it by EXEC or SP_EXECUTESQL. but rules may turn quite complex, for
example, audit log, or more complex issues, shall we do it in .net
program? but how can a trigger in SQL database informing a .net
program, or webservice, or windows service? by executeing an executable
console program?

3. user may raise more and more rules, how could we provied a flexible
solution? i mean we're trying to build it less hard-coded. seems in sql
database EXEC or SP_EXECUTESQL are still somewhat flexible, while in
..net to do something like eval() in javascript or EXEC in sql server is
a little bit troublesome. but, put all these bussiness logic in stored
procedure sounds a little bit weired.

guys, hope i have made myself clear. any suggestion? Thanks very mcuh

yours,
athos.

View 7 Replies


ADVERTISEMENT

REAL Project ETL Package Transaction Handling

Dec 9, 2007

Hello,
I am reading Microsoft REAL project ETL part. I find that it uses neither transaction nor checkpoints throughtout the packages. I am wondering if something goes wrong with the process and the process aborts in the middle,how are we going to handle this. Any particular reson not using transaction? Is it best practice without using transaction for performance consideration and log volumn? I am new to SSIS. I would like to follow the best practice. Thanks for advice..


Regards

Jerry

View 6 Replies View Related

Move Report File From Reporting Server Back To Local Visual Studio Project

Apr 4, 2008



I am not sure if this is possible or not, but I have to at least ask.

I have a SQL Reporting Services project that has a QA server version and then a Production server version. So when requested changes are made to the reports, I make the changes in Visual Studio and then push to QA. After the changes have been tested and approved, I then push the changes to the Production server.

I have come across an issue, where requested changes were made in Visual Studio (a lot of changes), and pushed to QA. The users now have changed their minds and they want instead to stay with what is in production. So, I am looking for a way to recover the report file on the production web server, bring it back into my visual studio project to replace the report that I had changed.

Is this possible, or will I have to start over and step one and reverse the changes in Visual Studio.

Let me know.

Thank you,

T.J.

View 3 Replies View Related

Use Report Server Project To Connect To Report Model Project Data Source View?

Jan 17, 2008

Hello,
I've created a Report Model Project that can be used by Report Builder to generate ad-hoc reports. I'm trying to create a connection string in my Report Server Project that points to the Report Model Project data source view.

All I can do is create a regular datasource, which bypasses the metadata contained in the Data Source View.

Basically I want my Report Server Project and my Report Builder reports to leverage the same metadata. Is this possible? If so how do I get the connection string?

Thanks!!
-Matt

View 1 Replies View Related

Making DATEDIFF Flexible

Feb 26, 2007

I have the following SQL statement that works out how many days are between todays date and the first of April 2007.

SELECTDATEDIFF(dd,'2007-04-01 00:00:00.000',GetDate()) AS 'Days Left'

This works fine, however: After the 1st April 2007 I want to start counting down the days till 1st April 2008 - and so on and so forth.

How can I do this? Hopefully you understand my question - if not, ask me any questions needed!

Cheers - GeorgeV

View 14 Replies View Related

Team Project In TFS - SSIS Project Sets Itself As The Startup Project

May 29, 2008



For some reason in a Team Foundation Team Project that has multiple project types (SSRS, SSIS, WebSite, C# Business DLL...), the SSIS project makes itself the startup project to the team project. If I explicitly set another project as the startup project to the team project and then select an SSIS package in the SSIS project in the team project, the SSIS project becomes the startup project automatically.

I am using Visual Studio 2005 SP1.

View 3 Replies View Related

Making A Query Have Flexible Index Utilization

Apr 29, 2008

I have one query that looks like this.

DECLARE @propname

SET @propname = NULL -- can be Null or have wild card values

SELECT col1, col2, col3, propertyname FROM testtable where col1 = 1 and col2 = 2 and col3 = 3 and (@propname IS NULL OR propertyname LIKE @propname)

col1, col2, col3 are part of a clustered index

propertyname is a nonclustered index

This is the predicament. If the "@propname is NULL" is first in the OR statement then the query will use the clustered index for finding the record. If I put "@propname is NULL" last then it uses the propertyname index no matter what. So I either get full index scans if NULL is ever used on property names or I get consistent two second long searches on the clustered index. Any way to have the best of both worlds? Or do I have to divide up my query into more stored procedures?

THANKS

View 11 Replies View Related

SQL 2012 :: Optimal Combination Of Tools For Flexible Reports

May 28, 2014

I would like to build a report with nice functionalities like filter, sorting, drill-down, something like a PowerPivot Table, but with some layout/design/format capabilities. I would also want to publish the report, refresh it let´s say once a week, notify users when a new version is available, etc.

If I use PowerPivot, then I am not able to customized the report or to mix data from different sources in one table.

If I convert the cells of the PowerPivot table to workbook formulas I lose the filter, sorting, etc functionalities.

I still have to try using Reporting Services, but I think that always something is missing.

View 1 Replies View Related

Handling Bad Data?

Jan 2, 2008

I have a peculiar requirement. I need the unmatched data to passed on a bad data table as well as tied to a dimension "Others". Please guide how to suffice these two condition simultaneously

View 1 Replies View Related

Handling Bad Data

Jul 25, 2007

I have an SSIS package that takes in a flat file and pushes the data into a table. However, once in a while the file will have some bad data in it (for example, this particular time I have too many delimiters on one line). I want the package to redirect the row to an error table and keep going with the processing for the rest of the data. To do this, I have hooked up an error output from the flat file source to an OLE DB destination and assigned all the rows to Redirect Row on Error in the Error Output section of the flat file source. Unfortunately, it does not work! The flat file receives an error and stops. Is this because something different has to happen when it is a problem with the whole row? Any help would be appreciated, thanks.

View 6 Replies View Related

MS SQL Server Handling Unicode Data

Feb 21, 2007

Hi,

I want to fetch Unicode data from MS SQL server 2005.I have created one user with russian language characters.

Following is my code snippet:
------------------------------------
wchar_twchar1[55];
char VALUE[255];

while (rc != SQL_NO_DATA)
{
rc = SQLFetch(hstmt);
if (rc != SQL_NO_DATA)
memset(VALUE,0,255);
int length=wcstombs( NULL,(wchar_t *)&wchar1, wcslen(wchar1));
char* strChar = (char *) malloc((length +1 )* sizeof(char));
if (strChar != NULL)
{
count++;
memset(strChar,0, length + 1 );
int i =wcstombs(strChar, wchar1, wcslen(wchar1) );
printf( "i is %d",i);
}
printf( "The length is %d",length);
wprintf(L" <%s>", wchar1);
printf(" simple char <%s>", strChar);
wprintf(L" simple char <%S>", wchar1);
}


-----------------------------------

CAn anybody help me ?
I think problem is in fetching of name.
Does anybody have working code for fetching unicode data from SQL server 2005.

Also I want query that will tell the locale setting of the server and also client?


Thanks-----

View 4 Replies View Related

Handling High Volume Data

Apr 18, 2007

Hi Guys,

I am facing problems with concurrent access in SQL Server 2000,The scenario is that the DB contains one huge de-normalized table containing 40 million records.

The application frequently queries this table to populate other derived tables,the sql queries take a long time to return results.

So if one query is in execution the other user's query goes into a
wait mode.Please suggest how I can better this.

Or do I need to upgrade to 2005.

Regards,
Prashant

View 2 Replies View Related

Handling Multiple Data Records With Sp

May 2, 2007

I have a need to read a table and extract records that matches a criteria (dough!, not that simple). Then I put the record in a text message and send them out via email. Works OK, but the problem is when I have multiple records for one entry. Since SQL doesn't have arrays "per say", I need to load the records on a "temporary array" then assembly the message at the end to include the records. Any suggestions how to "emulate" an array on a stored procedure (SQL), not sure what direction to follow. thanks

View 2 Replies View Related

Handling Encrypted Data In SSIS

Mar 11, 2008



Hi, I have source data in encrypted format. How should i handle it in SSIS?
I have found no information for such situation.

Anybody have any idea about it.

Bhakti

View 2 Replies View Related

Handling Log File During Data Import

Oct 3, 2007



Hi,
i have a package that uses a ForEach loop component to import flat files, and uses an OLE DB Destination component to insert the data into some staging tables (using table fast load with a max insert commit of 1000 rows), the biggest individual table import would be circa 5000 rows. At the end of each file import a stored proc is called to transfer the data into production tables, then the next file is imported.

Periodically (when importing more than 5 or 6 files) the process fails with the error message:

The transaction log for database 'blah' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases

This occurs when committing the data to staging. I do not start any transactions during this process, and the staging tables are cleaned out by truncating them, so i'm not sure exactly what is causing the log file to fill up (i'm not a DBA).

I know this is not specifically a SSIS problem, but can anyone give me a suggestion about the best way to handle the log file during an SSIS import? Should i execute a DBCC SHRINKFILE before each flat file is imported? Is there some other approach i should be taking to either insert the data or to move it from staging to production?

Thanks!






View 3 Replies View Related

Problem In Opening The Project..Make Sure The Application For The Project Type (.dtproj) Is Installed.

Jun 24, 2007

Hi,

I have been working on some SSIS packages for a while now and today while i was working i was trying to create a new connection and in the process there was an error and it said the BIDS has to be closed and i closed it but later when i open BIDS and try to open my project(.sln) from the file menu to work on the half done package it pops up an error which shows the path to my project location on the first line and next statement on the pop up error box says:



"Make Sure the application for the project type (.dtproj) is installed."



I tried to check some forums in which some suggested to try installing SP1 which i tried but ..i dont know why but the SP1 fails to install (i dont know if its causing problem becoz i already installed SP2 as i had some other problem before for which the cure was to install SP2).



Did anyone here face such a problem before ?

I'd really appreciate if the experts here can tell a cure for this problem.



thanks,

Ravi

View 4 Replies View Related

Visual Studio 2005 Automatically Sets SSIS Project As Startup Project

Feb 23, 2007

I have a Visual Studio 2005 solution which contains a Sql Server Integration Services (SSIS) project.

In this solution I have explicitly set a Web application project as startup project, but whenever I edit a DTS package within the SSIS project, VS automatically sets the SSIS project as startup project and the package I edit as startup object.

Needless to say, this may cause some really unwanted incidents when running the solution - thinking that you're about to run the Web application project (that was explicitly set as startup project), but instead, you run the edited package in the SSIS project.

Is there any way to avoid having the SSIS project automatically setting itself as startup project, any workaround here at all? :)

Thanks.

View 4 Replies View Related

Integration Services :: Importing Deployed Project From Catalog Into New SSIS Project

Jun 16, 2015

in order to maintain a deployed project into an Integration Services Catalog I'd like to know if it is possible to import it into a new project inside SSDT.

View 5 Replies View Related

PSI: How To Update Custom Field Value For A Perticular Project Using PSI(Project Server 2007)

May 6, 2007

Hi,



I want to update value of a custom field for a perticular project in Project Server 2007 using PSI.



I have created 5 enterprise custom fields(A,B,C,D,E) through PWA/Server Settings.



I want to search all Projects on Server. If any project is having value for custom field A then I want to update rest of the custom fields(B,C,D,E) for that perticular project.



I dont know how to do it please help.





Thanks in Advance



Madhukar

View 5 Replies View Related

Can't Create A Setup Project For Project Referencing Microsoft.SqlServer.ManagedDTS

Feb 3, 2007

I have a very small project written in VB.Net 2005 using the SQL Server 2005 SSiS DTSx package.

I took a SQL Server 2000 dts package and using the SQL Server 3005 legacy tools migrated it so I could still use the package withing SQL 2005 until I can build one using BI/SSIS.

Anyway,I added the reference Microsoft.SqlServer.ManagedDTS so I could then use the Microsoft.SqlServer.Dts.Runtime so I can execute the commands:

Dim oApp As New Application
Dim oPkg As New Package
oPkg = oApp.LoadPackage(g_DTSx_Directory & "AOC copy Generic1 CSV to AOC_verify_file_1.dtsx", Nothing)
Dim oResults As DTSExecResult
oResults = oPkg.Execute

Ok. That works fine. Executes without a hitch. So now I try and create a setup project for this and I use the setup wizard.

During the creation of the setup project I get a message that states:
The following files may have dependencies that cannot be determined automatically. Please confirm that all dependencies have been added to the project.
C:windowssystem32msxml6.dll

OK. The dll is part of the reference I mentioned above and I have no idea what other dependencies it may have.

How do I find this out?

Has anyone else created a project like this and experenced the same?

I am on a clean build running WinXP Pro with SP2 - VS2005 with SP1 and the SQL Server 2005 tools.

View 1 Replies View Related

Integration Services :: Automate Process Of Creating A Project On SSISDB Based On SSIS Project Name?

Apr 20, 2015

I've got an SSIS solution file with project deployment model in VS 2013 and would like to deploy that to SSISDB on different environments.All these days I followed the regular way to create a project in SSISDB and deploy it to that. Now want to find out if i can automate this process and so got some questions 

1. Can we automate the process of creating a project on SSISDB based on our SSIS project name?  This will be like when we do a deployment it should check if the project exists or not on SSISDB based on our SSIS project name, if the project exists we just deploy the packages in the project and if the project does not exists in SSISDB it will create that project and deploy the packages.

2. Can we also automate the process of creating environments? In traditional way we manually create the environment variables under environment tab of SSISDB, but can we make that also as part of deployment? Like when we are releasing to Dev server we look if that particular Dev variable exists on that server, if it exists we just update the existing stuff and if it does not exists we just create it.

View 2 Replies View Related

Business Intelligence Project/Integration Services Project Not Showing Up In Visual Studio 2008

Mar 31, 2008

I have visual studio 2005 and sql server 2005 with integration service installed on my machine. Couple of days ago, I installed visual studio 2008 professional. When I go to create SSIS project I dont see it in visual studio 2008. What do I have to do to make it appear in visual studio 2008 so that I can create SSIS projects.

Thanks.

View 7 Replies View Related

Unable To Use Project Professional To Establish Connection To Project Server

Jul 23, 2007

I fail to use project professional 2003 to access to the project server 2003 using MSDE 2000 in local area network, following message was shown,



Connection failed:


SQLState: '01000' SQL Server Error 1326 [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (Connect())
Connection failed:
SQLState '08001'
SQL Server Error: 17
[Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied.



I have seen these pages with similiar cases but can't help.

http://support.microsoft.com/kb/818047/

http://support.microsoft.com/kb/837653/en-us



Can anyone help me?? Thanks a lot!

View 1 Replies View Related

T-sql Data Type Conversion Error Handling

Jun 21, 2001

I'm trying to convert a column from a varchar(14) to float, some of the rows contain some non-numeric characters. I've tried to write a cursor to step through the records, run cast(col1 as float) on each record, then use if @@error <> 0 to capture an error but it doesn't work. It steps through fine but when the cast command fails it ends the execution of the script with "Error converting data type varchar to float."

How can i capture this error without having the script fail? Is there another way to get this done?

Thanks,

Jim

View 1 Replies View Related

Handling Large Data Values In Textboxes

Mar 28, 2007

Hi, Ive got a report using a List item that is vertically displaying the columns from a table. The problem I run into, is that some of the fields in this table contain large blocks of text where the users have entered comments and such.

I am using Textboxes to display this data.

So my report will look something like
-----
Field label 1 Field value 1
Field label 2 Field value 2
Field label 3


<white space>



<page break>

Field value 3 ---> this is a big block of text
Field label 4 Field value 4
etc
------
It appears as though the report attempts to keep the contents of each textbox together even if that means breaking onto an entirely new page to do this. I would prefer for the data to flow more natrually instead where the page breaks in the middle of the data being displayed should it be too large to fit on the page it started on.

-----
Field label 1 Field value 1
Field label 2 Field value 2
Field label 3 Field value 3 --- As much as can fit on this page

<page break>

Field value 3 ---> remaining data that broke over the page
Field label 4 Field value 4
etc
------

Any suggestions would be apprecaited.

View 3 Replies View Related

Flat Data Length Handling In IS Package

Apr 11, 2008



hello All,

I have a flat file which has a column (length of 24) . Whenever I have that column more the 24 char length, my package failed. I would like to know if there is any way if I can continue my package in this situation and send my bad row to the output table?

Any help will be appreciated.

Thanks

View 2 Replies View Related

Data Flow Inside For Each Loop - Error Handling

Jun 23, 2006

Hopefully this is an easy question:

Inside of a for each loop (looping through an ADO record set of objects to import) I have a data flow task (along with many other processes).... if the dataflow task suceeds I log success in a table. If it errors I want it to fail the dataflow task (which will fire off my Event Handler for that data flow and log the failure, email etc) BUT I want it to continue the loop - I can't seem to figure out how to get the data flow object not to fail the whole loop. If any other objects inside the foreach, other than the data flow, fail I would like the whole loop to fail. Also if possible (but not a requirement) I would like it to have a threshold where if the data flow fails X variable times it will fail the package.

I am having difficulty how to not fail the loop when the import data fails..... just looking for a simple "on error next" type logic for that specific object in the foreach but not the rest. Thanks in advance for the help/advice.

View 4 Replies View Related

Error Handling In OLEDB Source In Data Flow

Sep 11, 2007

I am trying to execute a SP like below in OLEDB source in data flow... and this statement include the insert stament ( row by row transaction).. I would like to creat an error hadling logic so that if the trasaction fail to insert the row then ignore that particular row then, move to the next row without stopping the whole process.. how can i do this?


exec usp_Inert_Registration_Episodes_Assessments

@Unique_ID=?,

@Gender_Cd=?,

@Birth_Date=?,

@Race_Ind=?,

@Ethnicity_Cd=?,

@Registration_Dt=? ,
--
--@Object_Key

View 16 Replies View Related

How To Write Sql Data Into Xml In Vb Project?

Jan 17, 2008

hi all,
I'm working on vb.net 2.0 with clr stored procedures. i've created a project from VB-->Database. there is a stored procedure in my sql server 2005. In the vb project i have to read the stored procedure and write it as xml . and i have to give this writed xml to clr stored procedure.. do any one know the code for this? if so please post here..

View 2 Replies View Related

Having Difficulty Setting Back Up To Back Up File Wihout Datetime Stamp SQL 2K

Apr 24, 2007

Hello,I'm trying to create a simple back up in the SQL Maintenance Plan that willmake a single back up copy of all database every night at 10 pm. I'd likethe previous nights file to be overwritten, so there will be only a singleback up file for each database (tape back up runs every night, so each daysback up will be saved on tape).Every night the maintenance plan makes a back up of all the databases to anew file with a datetime stamp, meaning the previous nights file stillexists. Even when I check "Remove files older than 22 hours" the previousnights file still exists. Is there any way to create a back up file withoutthe date time stamp so it overwrites the previous nights file?Thanks!Rick

View 5 Replies View Related

Mirroring :: Principal Database Get Role Back After Being Back On Line

May 14, 2015

New to Database Mirroring and I have a question about the Principal database server. I have a Database Mirroring setup configured for High-safety with automatic fail over mode using a witness.

When a fail over occurs because of a lost of communication between the principal and mirror, the mirror server takes on the roll of Principal. When communication is returned to the Principal server, at some point does the database that was the previous Principal database automatically go back to being the Principal server?

View 2 Replies View Related

Reporting Services :: Run Two Reports Back To Back Without Page Eject?

Jun 9, 2015

I need to run two reports each of A5 Size to run back to page and print on single A4 paper means in 1st half Sale bill will be printed and in second half Gate Pass Will Be Printed both report will be on same page and size and shape should be maintained. How to do it.

View 4 Replies View Related

T-SQL (SS2K8) :: Error Handling - String Or Binary Data Would Be Truncated

May 16, 2014

We have a wrapper procedure that i am looking at replacing shortly. Basic Layout is something like like

Create proc SP_Wrapper
as
SP_LogProc 'proc1','Start'
Exec proc1
SP_LogProc 'proc1','End'
SP_LogProc 'proc2','Start'
Exec proc2
SP_LogProc 'proc2','End'

SP_LogProc 'proc3','Start'
Exec proc3
SP_LogProc 'proc3','End'
....

You get the idea. So logproc just logs that the procedure has started or stopped to a table so we can monitor it easily.

The procedure is then run by an agent job every morning. This morning we had a little bit of an odd one. In the example above we effectively got to 'proc3' being started (as it was logged). However there was then an error of String or binary data would be truncated (severity 16 i believe). However when proc3 was then manually run it worked (the data was unchanged). Then going back proc2 works fine, and its actually proc1 that has the error.

I have looked through the procs in question (and the wrapper) and cant find any error handling that is relevent (there one try and catch block completely separate at the end of the wrapper procedure for a small routine).

View 5 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved