Sometimes it may take a minute or two for your Spark instance to initialize to get to the below screen. You should see something like below (ignore the error you see at the end). Spark-shell is a CLI utility that comes with Apache Spark distribution, open command prompt, go to cd %SPARK_HOME%/bin and type spark-shell command to run Apache Spark shell. Winutils are different for each Hadoop version hence download the right version based on your Spark vs Hadoop distribution from Apache Spark shell Winutils.exe enables Spark to use Windows-specific services including running shell commands on a Windows environment.ĭownload winutils.exe for Hadoop 3 and copy it to %SPARK_HOME%\bin folder. To run Apache Spark on Windows, you need winutils.exe as it uses POSIX like file access operations in Windows using Windows API. Many beginners think Apache Spark needs a Hadoop cluster installed to run but that’s not true, Spark can run on AWS by using S3, and Azure by using blob storage without Hadoop and HDFSe.t.c.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |