After some researching, I found that this is quite a common problem and most people suggest dynamically building the statement and that seems to work.
DECLARE @Cmd NVARCHAR(MAX) = 'BULK INSERT [TableName] FROM ''[InputFile]'' WITH (FIELDTERMINATOR = ''|'', ROWTERMINATOR = ''' +CHAR(10) +''')'
EXEC SP_executeSQL @cmd
However, I am not a fan of dynamic sql and tried to find out if it can be done without and ....woohoo! It can be - You just have to use the hex values for the row terminator and then the World falls back into a sensible orbit and life makes sense again.
BULK INSERT [TableName] FROM '[InputFile]' WITH (FIELDTERMINATOR = '|', ROWTERMINATOR = '0xa')
I hope this helps someone out there as it saved me a lot of time after I sussed it.