Order By User Defined Table Type

I've got a user defined table type called StringDictionaryTVP:

CREATE TYPE [dbo].[StringDictionaryTVP] AS TABLE(
[key] [varchar](500) NULL,
[value] [varchar](500) NULL
)

Ideally I would like to be able to have a # of columns and directions in this table like so:

DECLARE @OrderByClause StringDictionaryTVP

INSERT INTO @OrderByClause([key], [value])
values('gender','desc')
INSERT INTO @OrderByClause([key], [value])
values('name','asc')

Since our database can be a bit sizable, I'd also like to use Common Table Expressions so I can page through them fairly easy.So my standard cte is something like this:

DECLARE @PageIndex INT = 0
DECLARE @PageSize INT = 20
;WITH results_cte AS ( SELECT U.*, ROW_NUMBER() over ( ORDER BY name ) RowNum
from Users U)
SELECT * FROM results_cte
WHERE RowNum > @Offset AND RowNum <= @Offset + @PageSize

So where 'ORDER BY name' is I'd like to use the @OrderByClause in some sort of dynamic way. I've tried all kinds of stuff but even something like this doesn't get the actual column name I need

;WITH results_cte AS ( SELECT U.*, ROW_NUMBER() over ( ORDER BY (select top 1 [key] +' '+ [value] from @OrderByClause) ) RowNum
from Users U)

I may be chasing the wrong stick, but outside of dynamic sql, is something like this possible?