Continuous Nonstop Pumping Of Data
Mar 4, 2007
I have one data pump in a series that was pumping in too many records. Doing an independent query of the source table, I found there was about 140,000 records. My pump uses a variable for the source query, nothing fancy just a simple SELECT * FROM table WHERE DateField > '4/6/2006 12:00:00AM'. The Destination is local on the SQL Server and is set by a variable, and does a fast load. When I went away and checked in BIDS while it was running (the data flow tab where you can see the record count) it was at 28,000,000 and still going!
Any ideas what could be causing this? As I say there are only 140,000 records and no joins in the query--is this a bug someone has run into before?
Thanks,
Kayda
View 2 Replies
ADVERTISEMENT
Mar 30, 2007
Hi, there;
I created a package that pump data from ODBC to SQL2005.
When I run my package, I got error:
SSIS package "Package.dtsx" starting.
Information: 0x4004300A at Data Flow Task, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Data Flow Task, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Task, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x4004300C at Data Flow Task, DTS.Pipeline: Execute phase is beginning.
Error: 0xC02090F5 at Data Flow Task, DataReader Source [1350]: The component "DataReader Source" (1350) was unable to process the data.
Error: 0xC0047038 at Data Flow Task, DTS.Pipeline: The PrimeOutput method on component "DataReader Source" (1350) returned error code 0xC02090F5. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.
Error: 0xC0047021 at Data Flow Task, DTS.Pipeline: Thread "SourceThread0" has exited with error code 0xC0047038.
Error: 0xC0047039 at Data Flow Task, DTS.Pipeline: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.
Error: 0xC0047021 at Data Flow Task, DTS.Pipeline: Thread "WorkThread0" has exited with error code 0xC0047039.
Information: 0x40043008 at Data Flow Task, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DF at Data Flow Task, OLE DB Destination [1527]: The final commit for the data insertion has started.
Information: 0x402090E0 at Data Flow Task, OLE DB Destination [1527]: The final commit for the data insertion has ended.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "OLE DB Destination" (1527)" wrote 0 rows.
Task failed: Data Flow Task
SSIS package "Package.dtsx" finished: Success.
Did I do something wrong when I created this package?
Thanks.
View 7 Replies
View Related
Nov 22, 2006
Hi,
The time column type is SMALLDATETIME. Data is inserted every minute, but it isn't occured always.
When I display the contents, I want show every time(minute). If a specific time is skipped, it should be display as data zero.
Someone help?
View 1 Replies
View Related
Mar 21, 2008
Hi...My first post.
I'm currently creating a SSIS package that takes data from 3 unique databases. A SQL DB, FoxPro DB, and an Oracle DB. The data is pulled, cleansed and put into a single SQL 2005 table. The data is then pulled from this table every 15 minutes, formated in a given specification and uploaded to an ftp site. This part is done. My question is this:
This package needs to run around the clock, non-stop. How can package be set up to do this? It needs to pull data from the 3 DBs and put it in the common table, wait 15 minutes and do it again. Wait 15 more mintues and do it again. And so forth. A problem I'm having is I don't see a way to set up a SSIS package so that it runs around the clock.
On same premise, I have another issue. When I try to take data from the common table and there is nothing there, it causes an error. Is there some way that you can run a test like
SELECT * FROM _table_ WHERE is_sent = 0
if results == 0
{
wait 15 minutes and test again.
}
else if
{
write flat file, wait 15 minutes.
}
This has to be done in the Control Flow scope, so I can't use a conditional split. This is a pretty big deal as this needs to run around the clock. Thank you in advance for your assistance.
View 4 Replies
View Related
Oct 13, 2006
Does anyone out there have any ideas on the best way to connect in integration services to a nonstop sql database running on a tandem computer? We have some odbc drivers that have proven to be problematic and slow to use. What oledb products I can find apparently come with something that's around $36K per processor to buy. Thanks.
View 1 Replies
View Related
Jun 22, 2015
We're having some issues with where our backups write to, so I've been watching and monitoring the performance, when I noticed today that restore labelonly from disk has been running almost non stop for the past few hours.
The account running the query is the SQL Server's service account, and the program is "Microsoft SQL Server".
Every minute or so the SPID changes which made me think it was related to the transaction logs, the "restore labelonly" runs for as long as each database in the transaction log backup.
Example: Database A transaction log backup takes 1 minute and the SPID XX for restore labelonly runs 1 minute
Database B transaction log backup starts and there is a new SPID for restore labelonly.
I hope this makes sense because I normally don't see this restore labelonly running.
View 2 Replies
View Related
May 26, 2008
Hi,
does anyone know if it is possible to set an alert like that ?
if cpu is over 85% and stay for more then 1min continuous over 85%then alert.
THX
View 15 Replies
View Related
Jun 20, 2008
hai,
set dateformat dmycreate table tbl_sampemptable(employeeid int,StartDate datetime)
declare @employeeid intset @employeeid=1declare @startdate datetime,@enddate datetimewhile(@employeeid<=1000)begin set @startdate='01/05/2008' set @enddate='31/05/2008' while(@startdate<=@enddate) begin if(@employeeid<>1 and @startdate<>'02/05/2008') insert into tbl_sampemptable values (@employeeid,@startdate) else if(@employeeid=1) insert into tbl_sampemptable values (@employeeid,@startdate)z set @startdate=dateadd(day,1,@startdate)
endset @employeeid=@employeeid+1end
select * from tbl_sampemptabledrop table tbl_sampemptableset dateformat mdy
i have to select records depending on @count parameter to this table.Depending on this parameter value it should fetch sequential dates.For example
if @count=2then result should be like this,
EmployeeID FromDate ToDate
1 01/05/2008 02/05/2008
1 03/05/2008 04/05/2008
.
.
2 03/05/2008 04/05/2008 //note that here 01/05/2008 is not selected
because 02/05/2008 is missing
2 05/05/2008 06/05/2008
.
.
3 03/05/2008 04/05/2008 //note that here 01/05/2008 is not selected
because 02/05/2008 is missing
3 05/05/2008 06/05/2008
.
.
if @count=3 then result should be like this,
EmployeeID FromDate ToDate
1 01/05/2008 03/05/2008
1 04/05/2008 06/05/2008
.
.
2 03/05/2008 05/05/2008 //note that here 01/05/2008 is not selected
because 02/05/2008 is missing
2 06/05/2008 08/05/2008
.
.
3 03/05/2008 05/05/2008 //note that here 01/05/2008 is not selected
because 02/05/2008 is missing
3 06/05/2008 08/05/2008
.
. how can i do this.please help me.thanks in advance
View 8 Replies
View Related
Mar 7, 2008
I am fairly new to SQL 2005 and have taken over a migration project from 2000 to 2005 and one of our scheduled jobs seem to run forever, but do not have errors. This did not happen in the past so I was wondering if the agent settings for Replication Merge has the -Continuous parameter will the job ever complete or does it really run "continuously"?
View 1 Replies
View Related
Jun 6, 2006
Hi,
l've a series of day which record the date of an event. l would like to count the # of continuous days for the event. In this case, it would be 14/5, 15/5, 16/5, 17/5, 18/5, 19/5 and 20/5. Any idea to do this in SQL?
Date
-----
20/5
19/5
18/5
17/5
16/5
15/5
14/5
09/5
07/5
06/5
05/5
View 7 Replies
View Related
Oct 23, 2007
Hello,
I have a question regarding whether or not Data Mining can be utilized in a specific problem I have to solve.
Situation: I€™m going to simplify the problem by explaining it in terms of a €śpizza manufacturer€?. Suppose I wanted to predict the run minutes + downtime minutes (I use these to get an hourly rate: Pizzas/(run hrs + delay hrs) = Pizzas per hour) by looking at a set of input properties.
My properties could be something like the following:
# of Toppings
# of Special Pricing Stickers
Cardboard Box Indicator
Case Indicator (0 represents auto-casing, 1 represents putting in case by hand)
Machine Type (0 or 1€¦ 0 represents an older €“slower machine, 1 is newer)
Quantity of Run
(there could be up to 15 other properties that may or may not impact our rate)
Measured Values:
Run Minutes
Delay (down) minutes
Steps I€™ve Done So Far:
I€™ve created a couple different data mining models for this as I was unsure which one(s) to use. I checked the lift chart while feeding back in the original data set and my scatter plot appeared fairly inaccurate.
I've attempted to use Excel to create a linear regression, however my r squared value was always around .30. I decided to try to use SQL Server Data Mining to see if it could be something to help predict our accuracy better than a linear formula.
I've played with a couple different algorithms in Data Mining, and it appeared that none of them did exceptionally well with prediction. I even checked the lift chart using the same table as I used to train the model.
What algorithm(s) might work the best?
Can I reasonably expect a prediction within a fairly strict tolerance (I'm guessing the answer to this is: "yes, if your source data represents a consistent pattern")?
How can I best utilize Data Mining to give an answer like "historically, your run rate has been between these 2 values with a probability of X". I'm thinking I can utilize the predictprobability and stdev to some extent.
Any suggestions would be greatly appreciated.
If anyone needs further clarification, please let me know.
Thank you.
Regards,
Dan
View 2 Replies
View Related
Apr 27, 2007
I am having one question about discretization of continous attributes
values. How does it work? I need this information for my thesis. I have a
continous attribut, namely SKS, with range 0-20. When I use Microsoft
Decision Tree algorithm, this attribut split in SKS <= 18 and SKS > 18. I
want to know how does it find 18 as a number to split not the other.
One question again about Microsoft Decision Tree algorithm, about
Complexity_Penalty parameter. How does it affect the algorithm? For example,
if I set this value=0.1 what does it mean and how does it correspond with
growth tree?
Thanks a lot before for your kindness to answer my questions.. :-)
View 1 Replies
View Related
Sep 19, 2007
Hi All,
I was wondering if there was a way to specify a range when training a model to predict continuous variables. For instance, the predicted variable can only have a range of 1 - 10.
Thanks
View 1 Replies
View Related
Jan 29, 2002
Does anyone know exactly how to create a trace that runs continuously on a server and writes the data to a table? Now I know how to create a trace file with the profiler, but I want something set up so that I don't have to have the profiler running on the server all the time. As well as soemthing that will restart itself if the server is rebooted. I have been looking at these x(xp_trace.*) procedures. Is this the way to do it?
I have to trap login information in a table and have a scheduled job that runs once a month and look for specific data in the table and send out e-mails based on certain values.
I have written the procedure which does this I just need to know how to set up the trace so it runs in the background continuously.
Can anyone help?
Thanks............
View 1 Replies
View Related
Oct 4, 2012
I have a table with below data. Requirement is to replace all integers with continuous 6 or more occurrences with 'x'. Less than 6 occurrences should not be replaced.
create table t1(name varchar (100))
GO
INsert into t1
select '1234ABC123456XYZ1234567890ADS'
GO
INsert into t1
select 'cbv736456XYZ543534534545XLS'
GO
EXPECTED RESULT:
1234ABCxxxxxxXYZxxxxxxxxxxADS
cbvxxxxxxXYZxxxxxxxxxxxxXLS
drop table t1
-----Table Proc Index Performance TSQL &&%$#@*(#@$%.......------------
Deep Into SQL Jungle
View 9 Replies
View Related
Apr 11, 2007
I was wondering if anybody knows how to solve this problem. Here's the setup.
There is an ASP.NET application running on a local web server at the customer's location, it currently uses a MSDE backend database. There is a copy of the application on the customer's webhosting company so it can be accessed from outside the customer's location it is running on a full version of SQL Server 2000. We have this setup because the customer's ISP is not very reliable and the customer needs to be able to use the application even when their web connection is down. It is also used from outside their location by sales people and management and remote offices. The problem is we want to keep both databases synchronized together. We had been using Merge Replication which was working fine until the local ID jumped because it had run out of allocated identities. This causes a problem for their accounting because now there is a gap in the document's numbers.
Is there a way to have the identity field (or a generated document number) to remain continuous and unique across both databases? This needs to also work if one of the databases were to go down for a time or lose connectivity between the two servers. I'm looking for any option. We also have the option of upgrading the application to SQL Server 2005 if needed. Any ideas are appreciated.
Thanks,
Seth
View 6 Replies
View Related
Aug 22, 2006
Hi
I have set up merge replication and it works nicely.
I have set it up to work continuously, because I thought that if it can't find the subscriber or is offline then that's fine it will just sync again when it's back on line.
This is true
BUT it keeps throwing lots of messages into the event log to tell me the merge has failed.
SO
a. Can i just turn off the error reporting
or
b. How can I get it to sync this way automatically on connection without the error messages
thanks
ICW
View 1 Replies
View Related
Dec 18, 2007
It seems i face a problem with the Microsoft Decision Trees model when i have a predictable variable that is continuous.
I have created the whole model according to the AdventureWorks tutorial (and it informs me that the same procedure is followed with a continuous variable) and i have flagged the variable as continuous. Even though everything seems be going well, the results i get are not correct (after a cross check with another project already done and checked).
Is there something i am missing or i skipped while creating the model? Any suggestions that may help me are appreciated
Thank you in advance
View 1 Replies
View Related
Jan 17, 2008
hi all
how to find the continuous date from the given date range in sqlserver 2005
e.g.
2007-01-27 and 2007-02-02 and output should be
2007-01-27 2007-01-282007-01-292007-01-302007-01-312007-02-012007-02-02any suggestion?
View 1 Replies
View Related
Aug 11, 2015
I am trying to find the members who are having monthstartdate continuously for 11 months ;
here in my example 123 wont have monthstartdate continuesly for 11 months it has break for february '2014-02-01'; where as 222 and 223 has continus 11 months , so i need to pull such members .finding out the members continuesly(enrolled) having 11 months.
Below is the sample data i am referring.
memid    MonthStartDate
123 2014-01-01
123 2014-03-01
123 2014-04-01
123 2014-05-01
123 2014-06-01
123 2014-07-01
[code]....
View 28 Replies
View Related
Apr 23, 2015
I am getting SQL Time out exception after long run of 15-20 Hours, Please find the attachment for more details. My SQL queries  are not taking much time for execution , simple Update /Insert statements to local database.
Observed Activity manager also, looks fine. One more application connected to same database (Insert statements) works fine. In connection string I have modified Connection string ConnectTimeout = 2147483647; (max)
View 4 Replies
View Related
Jun 5, 2007
I have SQL 2000 Server that had a database called ABC and it has been moved to another server on 5-15-2007
I kept ABC database in Read_only mode for few days (just in case) on old server and finally dropped it on 5-20-2007 and I think I forgot to drop the associated logins. I started seeing login failure for user 'xyz' in error logs
When I first noticed the login failure error in SQL Server log for login xyz, I deleted xyz login but it did not stop the errors.
I have been trying from then and no luck in identifying the cause/ resolving this issue.
I have ran SQL profiler trace and caught the user hostnames, NTusername in few cases and Application Name and contacted the Application owner & user (who are in the trace) to stop windows service/ schedule jobs..anything that is pointing to old server but the bad luck is they are not aware of anything running or pointing to old server. The worst part is the user whose hostnames are shown in the trace have never used ABC database and do not have any idea.
Here is what I found in the profiler trace:
TextData LoginName NTUserName HostName ApplicationName
Login failed for user 'xyz' xyz AB00007 WAB000007 Microsoft (r) Windows Script Host
Today I have created the xyz login in the server and assigned model database with reader permission to see if log some different error but nothing new (the same login failure error)
Finally, I had no solution other than restoring the ABC database back to my old server and set it in to Read_Only mode to stop these errors and Now I see the login 'xyz' firing the query against the database
* Any hint or pointers why this is happening and any possible solution on this?
View 1 Replies
View Related
Aug 9, 2006
We have a table in our legacy database system representing health insurance polices. The customer can, and usually does, renew the policy after 12 months. The legacy database uses the renewal string "99/99/9999" to signify "continuous until cancelled", in other words, "forever", or until cancelled
We need to convert this legacy table into a sql 2005 table, which supports the concept of "forever". But how can we do this? ("99/99/999" is not a valid sql date type and we don't want to use varchar for dates.)
TIA,
barkingdog
View 1 Replies
View Related
Jun 26, 2014
Discuss the following sql query with respect to performance in an applicaiton involving more number of concurrent users creating and deleting records. The objective is to create continuous primary key integer values.
Table name: SitePage
Column DataType
--------- -----------
PageID BigINT
PageName nchar(10)
Query to insert new record
DECLARE @intFlag INT
SET @intFlag = 0
WHILE (@intFlag =0)
BEGIN
BEGIN TRY
[Code] ....
We don't want to use auto increment integer value for primary key because of the following reason
[URL] .....
We also don't want to use SEQUENCE as we have to create 50 sequence for 50 tables
We can't do trace flag 272
View 9 Replies
View Related
Aug 21, 2015
Paper is 21cm x 9cmPrinter is Epson LX-300..When I set this paper size, SSRS turns orientation to landscape and prints as if clockwise right rotated!I tried creating custom paper on print server options without success. I also tried setting the same paper size in Report Builder and Print Server but failed again.
View 3 Replies
View Related
Nov 6, 2007
I have a report that has one subreport. I am finding that if the entire content of the subreport will not fit within the space remaining on the page that it will not start displaying data from that subreport until the next page of the report, leaving a blank section in the report. I would like it to display as much as possible on the first page and then continue on subsequent pages.
Is there a way?
View 11 Replies
View Related
Nov 24, 2006
Hi, all here,
Thank you very much for your kind attention.
I am wondering if it is possible to use SSIS to sample data set to training set and test set directly to my data mining models without saving them somewhere as occupying too much space? Really need guidance for that.
Thank you very much in advance for any help.
With best regards,
Yours sincerely,
View 5 Replies
View Related
Dec 14, 2005
After testing out the application i write on the local pc. I deploy it to the webserver to test it out. I get this error.
System.Data.SqlClient.SqlException: The conversion of a char data type to a
datetime data type resulted in an out-of-range datetime value.
Notes: all pages that have this error either has a repeater or datagrid which load data when page loading.
At first I thought the problem is with the date, but then I can see
that some other pages that has datagrid ( that has a date field) work
just fine.
anyone having this problem before?? hopefully you guys can help.
Thanks,
View 4 Replies
View Related
Dec 4, 2007
I have used both data readers and data adapters(with datasets) in the projects that I have worked on. I am trying to get some clarification on when I should be using which one. I think I am doing this correctly but I want to be sure I am developing good habits.
As the name might suggest, it seems like a datareader is for only reading data. I have read that the data adapter and dataset are for a disconnected architecture. Or, that they can be used for this type of set up. I have been using the data adapter and datasets when writing to a database and the datareader when reading from a database.
Is this how these should be used? Is the data reader the best choice for reading data? Am I doing this the optimal way from a performance stand point?
......................................................thanks in advance
View 1 Replies
View Related
Nov 2, 2015
We already integrated different client data to MDS with MS Excel plugin, now we want to push back updated or new added record to source database. is it possible do using MDS? Do we have any background sync process to which automatically sync data to and from subscriber and MDS?
View 4 Replies
View Related
Oct 18, 2006
When I enter over 4000 chars in any ntext field in my SQL Server 2005 database (directly in the database and through the application) I get an error saying that the data could not be updated because string or binary data would be truncated.Has anyone ever seen this? I cannot figure out what is causing it, ntext should be able to hold a lot more data that this...
View 7 Replies
View Related
Aug 12, 2015
I have a requirement to implement CDC for 50+ tables to implement incremental data changes warehouse/reporting rather than exporting the whole table data. The largest table is having more than half a billion records.
The warehouse use a daily copy of OLTP db (daily DB refresh). How can I accomplish this. Is there a downside in implementing CDC just for the sake of taking incremental changes on the tables?
Is there any performance impact if we enable CDC on OLTP db?
Can we make use of the CDC tables on the environment we do daily db refresh so that the queries don't hit OLTP database?
What is the best way to implement CDC to take incremental changes for reporting.
View 0 Replies
View Related
Jul 20, 2005
Hi,This is driving me nuts, I have a table that stores notes regarding anoperation in an IMAGE data type field in MS SQL Server 2000.I can read and write no problem using Access using the StrConv function andI can Update the field correctly in T-SQL using:DECLARE @ptrval varbinary(16)SELECT @ptrval = TEXTPTR(BITS_data)FROM mytable_BINARY WHERE ID = 'RB215'WRITETEXT OPERATION_BINARY.BITS @ptrval 'My notes for this operation'However, I just can not seem to be able to convert back to text theinformation once it is stored using T-SQL.My selects keep returning bin data.How to do this! Thanks for your help.SD
View 1 Replies
View Related
