Machine Learning Research Conferences are Like the Stars Wars Force

In the Star Wars movies, the Force has a good side and a Dark Side. Machine Learning research conferences are the same — there’s good and bad.

The good sides of ML research conferences are obvious — advancing knowledge, fostering collaboration, and so on.

But there’s a Dark Side too. Here are four shaky intellectual integrity factors I’ve read about. Some of these criticisms I agree with, some I don’t.

* Research paper accept / reject decisions are mostly arbitrary. A rejected paper, if resubmitted, will be nearly as likely to be accepted as rejected. And an accepted paper, if resubmitted, will be nearly as likely to be rejected as accepted. So there’s very low correlation (close to zero) between actual research quality and conference paper accept-reject. See https://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext.

* For some attendees, the real purpose of attending a research conference is often to get a free vacation to an exotic location, and do a job hunt. These are legitimate goals and if explicitly stated there’d be no problem. But the Pinocchio fib is that, “I go to research conference XYZ to learn about new developments in ML.” Nonsense. All conference information is available on the Internet. See https://www.bbc.com/worklife/article/20130712-conference-shirking.

* Researchers consistently submit papers with the “least publishable unit” — a tiny change from an existing paper. This is done to increase chances of paper acceptance and bulk up the number of accepted papers on a researcher’s curriculum vitae. This has a huge negative effect of swamping ML with essentially meaningless papers. Additionally, truly novel and interesting ideas have a greatly reduced chance of being published. See https://en.wikipedia.org/wiki/Least_publishable_unit.

* A large number of ML research papers (probably the majority) are guilty of mathiness obfuscation. The introductory section of a research paper, in order to be accepted to most conferences, must have a lot of obscure, impressive-looking math equations. These equations are often not essential to the paper. The dishonesty here is that the purpose of these equations is to overwhelm reviewers (who are often inexperienced) and lead the reviewers to believe that the research is deep and meaningful. See https://queue.acm.org/detail.cfm?id=3328534.

Now, don’t get me wrong, research conferences are necessary and good for the most part. But everyone and everything has a dark side to some extent. The largest ML research conferences seem to be most susceptible to The Dark Side, but luckily there are small ML conferences that don’t seem to fall into the shaky integrity traps as often.

In general, machine learning industry conferences (as opposed to research conferences) don’t have these intellectual integrity issues. Industry conferences, like most things in business, always have an eye on return on investment. Most of the industry tech conferences I speak at emphasize education and training that can be applied quickly and supply measurable value.



Left: The most well known depiction of Pinocchio is the Disney version from the fantastic 1940 animated film. Second from left: This image is from the original 1883 book “The Adventures of Pinocchio” by author Carlo Collodi. And here are two other Pinocchio images I found on the Internet.

This entry was posted in Conferences, Machine Learning. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s