My students often say that English is really difficult, and at the same time people say that Japanese must be a really difficult language to learn. Which is harder? I guess there are several ways of looking at it. First, it's all relative. If you are a German or French speaker, English is probably pretty easy. We have the same alphabet, and lots of cognates. But if you are a speaker of Hindi, then it might be harder.
My guess is that there are lots of factors that would make a language easier or harder, but there are two scales that I have run into recently that rate languages harder or easier to learn. This list at Lexiophiles is a one that was created after a survey conducted by the British government. The list goes like this:
The next list is from the US Defence Language Institute for Language, and they list the languages like this, I being relatively easy, and IV being relatively difficult
I French, Italian, Portuguese, Spanish
III Hebrew Hindi, Farsi, Russian, Thai
IV Arabic, Chinese, Japanese, Korean
My guess is that this is a list made up for American speakers of English, so English doesn't appear in the list, and the languages that appear there would be difficult for Americans to learn. I don't think Korean would be that difficult for a Japanese person to learn, because they have more similarities than Japanese and English for example.
Anyway, it's all relative, this question of what is harder? If you are motivated and enjoying your learning, it will be a breeze. If you are slogging it out in a dull class, any language will be torture.