Leave on your last day as you said in your letter, start your new job. The company you left could of course sue you for breach of contract but As you are not really senior what would be the point. They won’t give you a bad reference either but probably wont give you a glowing one, so don’t ask for one. It might make getting other jobs harder, if asked your current employer might say you are unreliable, etc. But don’t let that stop you taking on your new job.