I have a situation where I have table with over a billion records and needs to be scrubbed. Table does have a field with date time timestamp. I have been deleting rows from the table using the script below which basically provides me delete statements by date for records older than 90 days.
But now on each day row count is over 30 million rows and it takes forever to delete by date and transaction log becomes humongous.
So I would like to scrub it in 5 minute intervals instead of daily for records older than 90 days. Even in 5 minute intervals the record count tends to be around a million. This will keep the delete slice small enough to not a gigantic transaction log.
declare @startdate Datetime
declare @enddate Datetime
set @startdate = getdate()-480
set @enddate = getdate()-90
WHILE (@startdate < @enddate)
BEGIN
print 'delete from vending where DetectedDate < ''' + CONVERT(varchar(10), @startdate, 101) +''''
set @startdate = @startdate+1
END
I am hoping to modify the script above to produce a script with statements like this for a window between last 90 and 120 days:
delete from vending where DetectedDate <'6/15/2015 8:25:00 PM'
go
delete from vending where DetectedDate <'6/15/2015 8:30:00 PM'
go
delete from vending where DetectedDate <'6/15/2015 8:35:00 PM'
go
I'm trying to create a query to return Open, Close, Max and Min Price for each 1 minute interval. Source data has two fields - Price, and Datestamp at 5 second intervals.
I can calculate the Max and Min (below) and set the datestamp to the middle of the interval, but get stuck on how to also return the Open and Close price for each interval.
SELECT MAX(price) AS MaxPrice, MIN(price) AS MinPrice, DATEADD(ss, 30, DATEADD(n,DATEDIFF n, '1/1/2006', DateStamp),'1/1/2006')) AS DateStamp FROM MasterData GROUP BY DATEDIFF(n, '1/1/2006',DateStamp)
I have two fields ID and Log data and log data is a 96 character long string of numbers representing 15 minute intervals from midnight to midnight.
I need to convert these 96 characters to a full 1440 characters which would mean taking each of the 96 characters one by one and making 1 character into 15.
I had Vb macro to do the conversion but now it's broken and I can't fix it. Getting it done in SQL would solve a lot of problems.
I then go from the 1440 fields and do log analysis like total time doing a specific activity but my query is dependent on having all 1440 characters.
Hello Probably a very simple problem, but im stumped. I have a table which gives the start-time and end-time of an employees work day. I want to create a view which contains a line of data for each 5 minute period worked. Please help.
I have collected perfmon data that is in every 15 seconds. I need to run a query that will only retrun rows that are 5 minutes from the last row starting at a specific date/time.
Table Name: EmployeeDetails Columns: EMpID - Date - WorkedHours
For each day I get details of number of hours worked by each employee in this table.
Now my HR wants a report with such columns
empid - Week - Month - Qtr
So, week will have Sum of hours worked by employee in that week Month will have Sum of hours worked by employee in that Month Qtr will have Sum of hours worked by employee in that Qtr
I have a table that I need to do some computations on all the data but first I need to remove the duplicate records and insert the results into a destination table. Here's the example below. My table has 3.1 million rows. I have tried using the DISTINCT and the GROUP BY but both ways to select the data takes about half a minute to run. I'm wondering if there is a way to increase performance. Users are ok with this time since the process runs overnight but improving it won't hurt. I do have a clustered index on these fields but that doesn't seem to improve any.
I have 1 table with a huge amount of data that I recive from someone else in a flat file format. I want to be able to filter through that data and scrub it and find out the good data and bad data from it.
I'm scrubbing the data using different stored procs that i've created and through a web interface that the user can pick which records they wish to create.
If I were to create a new table for clean records, what is the syntax to keep Appending to that table through the data that i'm obtainig via the stored procs that i've created.
Any thoughts or suggestions are greatly appriciated in advance
I have a SQL table that has data filled with million records and the date is in minutes it looks like :-
RowDateTime Meter 1 Meter 2 Meter 3
25/05/2006 02:49:00 1220 450 489
25/05/2006 02:50:00 1223 470 500
25/05/2006 02:51:00 1227 490 511
25/05/2006 02:52:00 1230 510 522
25/05/2006 02:53:00 1233.5 530 533
25/05/2006 02:54:00 1236.9 550 544
25/05/2006 02:55:00 1240.3 570 555
25/05/2006 02:56:00 1243.7 590 566
I want to make a query to the above table and convert the data to houlry by summing Meter1,Meter 2 and Meter 3 to be the average. I want to import all the hourly data to a new table that will look like :-
Just wondering what is the best time to ensure that we only return data when the datetime field is the same when compared between two datetimes within a minute difference.
As in the following should return the data:
'2015-04-09 09:00:20' compared to '2015-04-09 09:00:50'
And the following should not return the data:
'2015-04-09 09:01:20' compared to '2015-04-09 09:00:50'
The problem, is that I'm merging data from three different result sets, which they all have data for every minute, however, the timestamp can be different by seconds or milliseconds.
So, I'm only interested to return the data when the two fields that I'm comparing are equal within a minute. I need to ignore seconds and milliseconds.
When I use the DTS GUI and insert a "Bulk Insert Task" the main tab says:
"Import text files into SQL Server. You cannot validate, scrub, or transform data using this task".
So my question is, what shoud you use to validate and scrub?
In particular I have fixed-format text file with some occasional bad records (e.g. wrong length, empty record). What should I be using? If you suggest vbscript could you show me some examples? I'm new to vbscript.
Is that like some interval function for dates, like i want to group my data in intervals of 15 minutes and 30 minutes. Is there such a function in T-SQL
I am working on a line chart. The variable on the X- axis is DateTime. The requirement is to have the values displayed with an interval of 4 hours (or mabe 5 or 6 hours - basically every regular intervals). I am using a list control to Subreports to show around 10 reports.
HOW can I do this? The Major and MInor intervals only help in separating which samples will be displayed. But we cannot configure for regular intervals. Please help.
Ok, I know that there is a very smart programmer out there that can resovle my issue.
I am trying to calculate time worked by 15 minute intervals.
Example: Emp 1 started work at 13:00:00 and worked 183 minutes Emp 2 started work at 17:15:00 and worked 150 minutes Emp 3 started work at 08:30:00 and worked 17 minutes
I have a client that collects data from a manufacturing facility a one minute intervals. I already have sql statments to convert the 1 minute data to other timeframes (e.g. 30 min, 60 min, daily). However, now the client wants to look at data converted to irregular time intervals. For example, instead of looking at the first, second, third, etc. 60 minutes of a day, they wish to see data grouped irregularly: first 30 minutes, next 1 hr & 45 min, next 2 hours, next 1 hr & 30 min, etc. These irregular intervals could change; they may later want to look at the first hr, next 2 1/2 hrs, next 1/2 hour, etc.; or whatever strikes their fancy.
So far, all I've come up with is run one query for each desired time session and then do a join on all the resulting tables. Anybody have a better idea on how to do this?
Hi, I want to run a stored procedure automatically at regular intervals.The time duration is given by me as input details.for example i will give the date and i should send email to some recipients just one day before the given date.I want this to be done in sql stored procedures.
how can I get the time differences between them.Let's say , a person who click on break @ 12:00 PM and he is back and select I am back option @ 12:15 the total break time is 15 minutes. However, can I display this difference of break time.
ID Date Time Length Interval_Num 1 10/11/2014 9:00 420 14
Basically, length represents the # of minutes a person is scheduled for. We have 30 minute intervals. Interval_Num = (Length/30). I need the data to show each interval. For the above example it should only be 14 intervals.For the above example the solution should look like....
ID Date Time Length 1 10/11/2014 9:00 30 1 10/11/2014 9:30 30 1 10/11/2014 10:00 30 1 10/11/2014 10:30 30
In a report there are following in parameters: from date, to date, from time and to time.
I need to match these search criteria€™s against start time and stop time in assignments. The trick: How to write SQL that meets all different cases that can arise.
The query need's to traverse through every assignment, and see if its start and stop time is in the interval given by the user.
It should then return the minutes from the assignment that is within the interval.
I'm thinking of packaging the minute "extracting" logic in a Scalar-valued function.
It should then take the in parameters from the user, and start time and stop time from the assignment as arguments.
To consider: Crossing midnight.
E.g. the user puts in
From date: 2008-05-24
From time: 22:00:00
To date: 2008-05-26
To time: 02:00:00
Since the user wants to evaluate a time span that crosses midnight,
I'll have to look in to time spans like:
2008-05-24 22:00:00 - 2008-05-24 23:59:59
2008-05-25 00:00:00 - 2008-05-25 02:00:00
2008-05-25 22:00:00 - 2008-05-25 23:59:59
2008-05-26 00:00:00 - 2008-05-26 02:00:00
Furthermore the start- and stop time can be partially in the above interval.
E.g. an assignment starts 2008-05-24 21:46:11 and stops 2008-05-24 22:36:05.
This would then yield from the time spans above 36 minutes.
Is it possible expire a report cache after less than one minute? I'm looking for a way to only have a report hit the database once every 10 seconds, no matter how many people are hitting it. Thanks.
I have this table of Marks as shown below. All I need is to find the average Marks at various intervals of S.no. That is I need averages at every 3rd S.No. as shown.
S.No. Marks 1 ------ 5 2 ------ 5 3 ------ 6 1st Average Value here (16/3) 4 ------ 5 5 ------ 6 6 ------ 7 2nd Average Value here (18/3) 7 ------ 7 8 ------ 7 9 ------ 8 3rd Average Value here (22/3) 10 ----- 8 11 ----- 9 12 ----- 8 4th Average Value here (26/3)
So basically I need a new table which will have 4 average values for the table above. Of-course the table can be much bigger and the average values can be at any nth value of S.No.