JOIN Onto Itself And Contains The Lower Date

Hello,
I have been having a hard time with this issue. I am attempting to join a table onto itself to get the closest date onto a single row.
What i mean is:
I have the following data
id date
1 10/07/08
2 10/06/07
3 10/06/03
4 10/06/03

the new table should have the current id and the one closes to it as so.
1 10/07/08 2 10/06/07
2 10/06/07 3 10/06/03
3 10/06/03 null null
4 10/06/03 null null
but i am getting duplicates do to the 10/06/03.
1 10/07/08 2 10/06/07
2 10/06/07 3 10/06/03
2 10/06/07 4 10/06/03
3 10/06/03 null null
4 10/06/03 null null
i want so that if there is a duplicate i can take the id thats higher. I cant figure it out.
This is my current sql:

SELECT PB.ID,PB.StartDate, PB2.ID, PB2.Startdate
from table PB
left outer join table PB2 on PB.keyID = PB2.keyID
and PB2.StartDate < PB.StartDate
and PB.StartDate = (select top(1) StartDate from table PB3 where PB.keyID = PB3.keyID
and PB2.StartDate < PB3.StartDate order by PB3.StartDate asc)


Thanks for the help.