The game can handle the resolution as it works on NVidia 1080ti card, but you ran into problems with your card. Although it always starts in window mode for me, I can change it to borderless. My problem with the cryengine is that there is fixed limit on the maximum amount of pixels that the screen can have. For me on bigger resolutions than 1920x1200, some of the high quality options, f.e. shadow are grayed out in the options. This is an known bug that was later fixed. There is a fix for this but that requires injecting into the cryengine.dll which is not allowed with Anti cheat and you need access to the aion.bin parameters (https://github.com/zzsort/Aion-Version-Dll) Atm, I don't want to play on 1080 to have everything on, I don't even own a 1080 / 1200 monitor.
To come back to your problem. The resolution and settings of the cryengine are mostly handled in the system.cfg file. This file is encrypted, but there are editors out there that allow you to edit it and save it (just goolge it, dps meters require the system.cfg to be altered to work). With some tweaking, you might be able to get it to work, but playing in fullscreen window isn't that bad, I have more of a problem with the loss of the quality settings. If you edit the sytem.cfg or SystemOptionGraphics.cfg, make sure you turn on the read only flag on the file, the game will overwrite it. And makes copies, as the launcher might remove them / remove the read only lock, so you can put them back.