{"id":6259,"date":"2017-06-06T22:31:06","date_gmt":"2017-06-06T22:31:06","guid":{"rendered":"https:\/\/rsoftsolution.com\/blog\/databricks-releases-serverless-platform-for-apache-spark-along-with-new-library-supporting-deep-learning.html"},"modified":"2017-06-06T22:31:06","modified_gmt":"2017-06-06T22:31:06","slug":"databricks-releases-serverless-platform-for-apache-spark-along-with-new-library-supporting-deep-learning","status":"publish","type":"post","link":"https:\/\/www.rbsitsoftwaresolution.in\/blog\/databricks-releases-serverless-platform-for-apache-spark-along-with-new-library-supporting-deep-learning.html","title":{"rendered":"Databricks\u00a0releases serverless platform for Apache Spark along with new library supporting deep learning"},"content":{"rendered":"


\n<\/p>\n

Today to kick off Spark Summit, Databricks<\/a>\u00a0announced a Serverless Platform for\u00a0Apache Spark \u2014 welcome news for developers looking to reduce time spent on cluster management. The move to simplify developer experiences is set to be a major theme of the event overall. In addition to Serverless,\u00a0the company also introduced\u00a0Deep Learning Pipelines, a library that makes it easy to mix deep learning frameworks with Spark.<\/p>\n

If you haven\u2019t been following the latest developments in cloud-based data processing, Databricks is the commercial manifestation of the open-source Apache Spark project. The company\u2019s engineers spend their days building tools to support the Spark ecosystem, like those being announced today.<\/p>\n

As data becomes a larger part of decision making within large enterprises, new users are facing the daunting challenge of dealing with data pipelines and cloud infrastructure. Counter to what you might think, serverless doesn\u2019t literally mean that data manipulation occurs without servers. It merely means that users can accomplish tasks by drawing from one managed pool of computing resources\u00a0\u2014 alleviating the need for every user to have to perform low-level tinkering.<\/p>\n

\u201cSQL is stateless so it isn\u2019t hard to work with,\u00a0but making data science serverless is hard because it has states,\u201d Databricks\u00a0CEO Ali Ghodsi explained to me in an interview.<\/p>\n

If Serverless is Databricks attempt at breadth, Deep Learning Pipelines is the company\u2019s attempt at depth. I still wouldn\u2019t say that TensorFlow and other comparative deep learning frameworks are \u201ceasy\u201d to use, but they\u2019re a heck of a lot easier to use than LISP. This has made deep learning part of an increasing number of workflows, even if its use isn\u2019t quite commonplace yet.<\/p>\n

\u201cIf you want to distribute TensorFlow, you have to construct graphs manually and direct what goes to what machine,\u201d added Ghodsi. \u201cThat\u2019s really hard if you want to run on 100 machines.\u201d<\/p>\n

Databricks\u2019 new open-source library enables developers to convert deep learning models into SQL functions. Users can perform transfer learning with\u00a0Spark MLlib Pipelines and reap the benefits of\u00a0distributing computing with Spark.<\/p>\n

Lastly,\u00a0Ghodsi noted that Databricks\u2019\u00a0Structured Streaming is now generally available. The API supports the processing of sequential data streams. The company says that it prioritized minimizing latency during the\u00a0Structured Streaming development process. This ultimately has both cost and speed implications for customers dealing with problems like\u00a0anomaly detection.<\/p>\n

<\/p>\n

Featured Image: Bob Stefko\/Getty Images<\/small><\/p>\n

Article source: http:\/\/feedproxy.google.com\/~r\/techcrunchIt\/~3\/QU4e0aTFkkA\/<\/p>","protected":false},"excerpt":{"rendered":"

Today to kick off Spark Summit, Databricks\u00a0announced a Serverless Platform for\u00a0Apache Spark \u2014 welcome news for developers looking to reduce time spent on cluster management. The move to simplify developer <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[19],"tags":[25],"class_list":["post-6259","post","type-post","status-publish","format-standard","hentry","category-enterprise-software","tag-enterprise"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t