I was thinking about this recently...
And by "most difficult thing", I mean the one thing that you won't ever learn from a textbook, the type of thing that you'd need to spend years living in Japan and using Japanese all day every day to master.
Any ideas?
As for English, I think the use of prepositions is very difficult for Japanese people, since Japanese doesn't have it (it's also really hard to explain!)