\n
![](\"https:\/\/i0.wp.com\/rsoftsolution.com\/blog\/wp-content\/plugins\/RSSPoster_PRO\/cache\/63c89_gettyimages-166272733.jpg?w=640&ssl=1\")
<\/p>\n
\n<\/p>\n
Two months ago, Facebook\u2019s AI Research Lab (FAIR) published some impressive training times for massively distributed visual recognition models. Today IBM is firing back with some numbers of its own. IBM\u2019s research groups says it was able to train ResNet-50 for 1k classes in 50 minutes across 256 GPUs \u2014 which is effectively just the polite way of saying \u201cmy model trains faster than your model.\u201d Facebook noted that with Caffe2 it was able to train a similar ResNet-50 model in one hour on 256 GPUs using an 8k mini-batch approach.<\/a><\/p>\n This would be a natural moment to question why any of this matters in the first place. Distributed processing is a big sub-field of AI research but it\u2019s also quite arcane. Computing jobs are often so big for deep learning problems that they are most efficiently handled across a large number of GPUs instead of just a single GPU.<\/p>\n But as you add more GPUs, training time doesn\u2019t naturally scale down. For example, you might assume that if it took two minutes to train with one GPU it would take one minute to train with two GPUs. In the real world it doesn\u2019t work like this because there is some cost to splitting up and recombining complex quantitative operations.<\/p>\n What IBM is promising is the most efficient distributed deep learning library for breaking up a giant deep learning problem into hundreds of smaller deep learning problems. This all might seem petty in the context of a single compute job, but remember that companies like IBM and Facebook are training models all day every day for millions of customers. Every major tech company has a stake in this but it\u2019s often tough to compare results companies promise because of the sheer number of variables in any research effort.<\/p>\n Now you would be right to question the future meaningfulness of obsessing on incremental increases in distributed scaling efficiency \u2014 and you\u2019d be right. Hillery Hunter, director of systems acceleration and memory at IBM Research tells me that everyone is getting really close to optimal.<\/p>\n \u201cYou have gotten about as much as you can out of the system and so we believe we are close to optimal. The question is really the rate at which we keep seeing improvements and whether we are still going to see improvements in the overall learning times.\u201d<\/p>\n IBM didn\u2019t stop with just the ResNet-50 results. The company continued the work testing distributed training on ResNet-101, a much larger and more complex visual recognition model. The team says that it was able to train ResNet-101 on the ImageNet-22k data set with 256 GPUs in seven hours, a fairly impressive time for the challenge.<\/p>\n \u201cThis also benefits folks running on smaller systems,\u201d Hunter added.\u201dYou don\u2019t need 256 GPUs and 64 systems to get the benefits.\u201d<\/p>\n The deep learning library plays well with the major open-source deep learning frameworks including TensorFlow, Caffe and Torch. Everything will be available via PowerAI if you want to try things out for yourself.<\/p>\n <\/p>\n Featured Image: Martin Barraud\/Getty Images<\/small><\/p>\n Article source: http:\/\/feedproxy.google.com\/~r\/techcrunchIt\/~3\/rPukATCy39k\/<\/p>","protected":false},"excerpt":{"rendered":" Two months ago, Facebook\u2019s AI Research Lab (FAIR) published some impressive training times for massively distributed visual recognition models. Today IBM is firing back with some numbers of its own. <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[19],"tags":[25],"class_list":["post-7954","post","type-post","status-publish","format-standard","hentry","category-enterprise-software","tag-enterprise"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t