When it comes to using super new research models or methods in deep learning, implementing it yourself instead of looking for other implementations is almost a requirement. In my experience the majority of online "implementations" of research deep learning models or methods have subtle bugs, flaws, or outright implemented something different than the paper they reference. This isn't limited to random github code: for example, I wouldn't trust anything in tensorflow.contrib unless I read the source.
Having implemented algorithms from research papers I came to the conclusion that the same is true about the papers. It’s unbelievable how bad the quality of the description of algorithms often is.
Are you generally able to achieve the results the new research claims after you implement it yourself per their description?
I would have thought if their description is shit then their results are shit too (cherry-picked from hundreds of runs, outright fabricated, misinterpreted, incorrectly graphed, whatever - just plain doesn't work, like a bad recipe.)
As a followup if this is the case then how do you know what new deep learning research is worth your time to implement?
Completely agree. Well, tf.contrib I am usually happy with the docs and a glance through GitHub issues, but, yes, code found elsewhere for very new, or even just recent papers can be really hit or miss, and not always obviously so. There is a lot of great stuff out there, but sometimes even popular papers over a year old can still be a challenge to find good code for. finding 3 separate implementations, all stlighty wrong or unusable is not uncommon.