An "open problem" is a question without a known solution, and the interesting thing about user experience design is that we don't have a lot of open problems or questions, it's that we often have TOO MANY ANSWERS.
A couple of examples:
What is the optimal placement of a button in a UI? There are many different answers to that question based on the use of the control, the context of use, the device and screen size the control is rendered on, etc.
What screen size is best for reading? Same problem – you have to look at factors like visual acuity, pixel density, device design, etc. Any answer you define may be "correct" in some ways and context, and completely wrong in others.
Since user experience design is focused on creating effective solutions for users, we also have many different answers for user-centered questions. How do users accomplish the task we are designing a solution for? The answer varies based on the individual, though there are always common approaches that people take.
Any true open questions in UX must be questions that have no real answer, such as "What is the user thinking?" There's no way to have a definitive answer to such a question because, well, we aren't mind readers.