First of all, sorry for the late response! I actually didn’t know this AMA thread was still going, but I am happy to continue answering questions as long as people are asking.
That’s an interesting question, and it’s actually more complicated than it looks at first glance. The simple answer is that there are lists of “hardest languages for English speakers” all over the internet (some examples can be found here and here), and these lists probably do give a pretty decent idea of widely-spoken languages that are, overall, difficult for English speakers: for example, as noted in the babbel article, Mandarin Chinese, which is often named as one of the hardest languages, is a tonal language, meaning that what sounds like the exact same word to an English speaker could actually be three or four different words depending on the sort of “melody” (tone) you say the word with. Since English doesn’t use tone, it can be hard for English speakers to keep track of.
The longer response, however, is that it’s just a bit difficult to say. For example, it’s unclear exactly how these rankings are generated, and in some cases, they’re based on things that aren’t technically related to language itself. For example, one of the reasons Mandarin is often cited as being difficult is the writing system, which is character-based instead of expressing rough correspondences between sounds and letters, like English. And while you might want learn the writing system if you were learning Mandarin, strictly speaking, the writing system isn’t part of the language. You could still theoretically learn Mandarin without learning the writing system (e.g. by sound alone, or by learning Pinyin, a different writing system which works more like written English).
The other thing to consider is that these lists typically only consider languages that are really popular to learn or that are spoken by lots of people. That’s why you’ll repeatedly see languages like Mandarin, Japanese, and Arabic, but you’ll probably never see languages like Cree or Mohawk (North American indigenous languages) on those lists, even though they’re actually even more different from English than Mandarin, Japanese, Arabic, or most other languages you’ll see on those lists. They’re referred to as “polysynthetic” languages, which means that the language can basically express the entire sentence in a single word, a concept that’s pretty unheard of for most English speakers. On the other hand, fewer people are likely to try to learn these, so they don’t usually make it on these lists.
Finally, different things are difficult for different people. For example, although tone is notoriously difficult for English speakers, someone with musical training might actually not find it very difficult at all. And if you have a lot of interest in the culture, that might be a lot easier than trying to learn a theoretically ‘easier’ language you don’t really want to learn.
TL;DR: “Hardest languages for English Speakers” type lists are decent guesses at what languages are hardest for English speakers, and Mandarin is a reasonable choice for being one of the most difficult widely-spoken languages (for English speakers). But if you’re really interested in just learning a language that’s very different from English, look up polysynthetic languages and try learning one of those!