Sharepoint MVC 5 Azure "You do not have permission to view this directory or page."

I'm creating an App for SharePoint and it runs locally (in debug perfectly fine).

I publish the app to Azure and it behaves as expected when navigating to it directly (Unknown user error page), then I package the upload it to SharePoint, install/update and click into it. This is the only text on the page:

You do not have permission to view this directory or page.

I can remove the [SharePointContextFilter] and everything works, obviously I need this to access SharePoint data though...

In my SharePoint App I have FullControl permissions for SiteCollection and Web.

I have no idea what is causing this error, how to diagnose it any further or how fix it. Can anyone shed some light into the matter?