Select ROWS In Table A That Aren't In Table B

I've been scratching my head over this one.
It should be pretty easy to do, but I keep going over the same problem.

I have 3 tables (Suppliers, Categories & Suppliers_To_Categories).

Suppliers_To_Categories contains which suppliers are related to which categories.

The set-up is so (simplified):

Suppliers
id
1 Supplier 1
2 Supplier 2
3 Supplier 3
4 Supplier 4

Categories
id
1 Category A
2 Category B
3 Category C
4 Category D

Suppliers_To_Categories
id supplierId categoryId
1 1 1
1 1 2
1 1 3
2 2 3
3 3 1
3 3 3


What I am after is an SQL statement which tells me for a particular category, which suppliers ARE NOT related to it, so they can be assigned.
E.g.

Category D has Suppliers 1-4 unassigned
Category C has Supplier 4 unassigned
Category B has 2, 3, 4 unassigned
Category A has Supplier 2, 4 unassigned etc...

I've got the other side of the SQL statement which tells me which suppliers are assigned to a category (see below):
SELECT Suppliers.*
FROM CatToSupplier INNER JOIN
Suppliers ON CatToSupplier.supplierId = Suppliers.supplierId
WHERE (CatToSupplier.CatId = @CatId)

This is probably really easy, but I've become unstuck!