{"id":24911,"date":"2021-04-13T23:10:04","date_gmt":"2021-04-14T03:10:04","guid":{"rendered":"http:\/\/bangla.salearningschool.com\/recent-posts\/pyspark-development-environment-root\/"},"modified":"2021-04-13T23:10:04","modified_gmt":"2021-04-14T03:10:04","slug":"pyspark-development-environment-root","status":"publish","type":"post","link":"http:\/\/bangla.sitestree.com\/?p=24911","title":{"rendered":"Pyspark Development Environment #Root"},"content":{"rendered":"

Using Hortonworks HDP:
\n[https:\/\/github.com\/sayedjustetc\/TechnicalArticlesAndCode\/blob\/Pyspark\/pyspark-development-environment]
\n–comes with pyspark
\nHowever, default python version for Pyspark is 2.7
\nTo change Python version for Pyspark use:
\nexport PYSPARK_PYTHON=’\/usr\/bin\/python3.6′
\nTo make the export permanent, change it on .bash_profile file
\nvi .bash_profile
\nthe put the folowing line on .bash_profile
\nexport PYSPARK_PYTHON=’\/usr\/bin\/python3.6′
\n—-
\nyour HDP by default will not have Python3.6. Install as below:<\/p>\n

To install Python3.6, you can use<\/p>\n

yum install python36 [or similar will work]<\/p>\n

If it does not work, first try
\nyum upgrade
\nthe
\nyum -y install python36
\nor
\nyum -y install python36*<\/p>\n

—
\nYou can try to change your System python to 3.6 by using [soft link, verify the order of paths in the command]
\nln -fs \/usr\/bin\/python3.6 \/usr\/bin\/python
\n[ref: https:\/\/cmdlinetips.com\/2011\/07\/how-to-create-a-soft-link-to-a-directory-in-linuxmac-os-x\/]
\n—<\/p>\n

If you want to use packages such as numpy or networkx inside pyspark, you will need to install them.
\npip install numpy
\npython3.6 -m pip install numpy
\nor similar might\/should work.<\/p>\n

By default HDP does not come with pip or numpy or networkx installed.
\nYou will need to install pip first
\nyum install pip
\nyum install python-pip
\nyum install python36-pip
\nyum install python27-pip
\nor similar will work
\n—-
\nthen you can use pip to install numpy or networkx or similar
\n—<\/p>\n

For installing numpy or networkx or scipy or similar libraries on HDP and to work inside Pyspark,
\nYou can also get : get-pip.py first then use this to install pip<\/p>\n

wget https:\/\/bootstrap.pypa.io\/get-pip.py
\npython3.6 get-pip.py
\npython3.6 -m pip install numpy<\/p>\n

—
\nYou might need to run the following, if you see Python issues while installing numpy
\nyum install python36-devel<\/p>\n

[Reference: https:\/\/stackoverflow.com\/questions\/17443354\/install-numpy-on-python3-3-install-pip-for-python3] From: http:\/\/sitestree.com\/?p=12341
Categories:Root
Tags:
Post Data:2018-12-09 23:43:33<\/p>\n

\t\tShop Online: https:\/\/www.ShopForSoul.com\/<\/a>
\n\t\t(Big Data, Cloud, Security, Machine Learning): Courses: http:\/\/Training.SitesTree.com<\/a>
\n\t\tIn Bengali: http:\/\/Bangla.SaLearningSchool.com<\/a>
\n\t\thttp:\/\/SitesTree.com<\/a>
\n\t\t8112223 Canada Inc.\/JustEtc: http:\/\/JustEtc.net (Software\/Web\/Mobile\/Big-Data\/Machine Learning) <\/a>
\n\t\tShop Online: https:\/\/www.ShopForSoul.com\/<\/a>
\n\t\tMedium: https:\/\/medium.com\/@SayedAhmedCanada <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Using Hortonworks HDP: [https:\/\/github.com\/sayedjustetc\/TechnicalArticlesAndCode\/blob\/Pyspark\/pyspark-development-environment] –comes with pyspark However, default python version for Pyspark is 2.7 To change Python version for Pyspark use: export PYSPARK_PYTHON=’\/usr\/bin\/python3.6′ To make the export permanent, change it on .bash_profile file vi .bash_profile the put the folowing line on .bash_profile export PYSPARK_PYTHON=’\/usr\/bin\/python3.6′ —- your HDP by default will not have Python3.6. Install … <\/p>\n

Continue reading<\/a><\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1917],"tags":[],"class_list":["post-24911","post","type-post","status-publish","format-standard","hentry","category-fromsitestree-com","item-wrap"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":65780,"url":"http:\/\/bangla.sitestree.com\/?p=65780","url_meta":{"origin":24911,"position":0},"title":"Configure Mac OsX for Laravel Development with Valet #Laravel","author":"Author-Check- Article-or-Video","date":"July 14, 2021","format":false,"excerpt":"\/usr\/bin\/ruby -e \"$(curl -fsSL https:\/\/raw.githubusercontent.com\/Homebrew\/install\/master\/install)\" brew update brew install homebrew\/php\/php71 composer global require laravel\/valet export PATH=$PATH:~\/.composer\/vendor\/bin --- not required curl -sS https:\/\/getcomposer.org\/installer | php curl https:\/\/getcomposer.org\/installer | php --- php -r \"copy('https:\/\/getcomposer.org\/installer', 'composer-setup.php');\" php -r \"if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt';\u2026","rel":"","context":"In "FromSitesTree.com"","block_context":{"text":"FromSitesTree.com","link":"http:\/\/bangla.sitestree.com\/?cat=1917"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11070,"url":"http:\/\/bangla.sitestree.com\/?p=11070","url_meta":{"origin":24911,"position":1},"title":"\u09aa\u09be\u0987\u09a5\u09a8 \u09aa\u09cd\u09b0\u09cb\u0997\u09cd\u09b0\u09be\u09ae\u09bf\u0982 \u099f\u09bf\u0989\u099f\u09cb\u09b0\u09bf\u09af\u09bc\u09be\u09b2 : \u0987\u09a8\u09cd\u09b8\u099f\u09b2\u09c7\u09b6\u09a8 (Python – Environment Setup in bangla)","author":"","date":"October 20, 2015","format":false,"excerpt":"\u09b2\u09bf\u0996\u09c7\u099b\u09c7\u09a8: \u0986\u09ac\u09b0\u09be\u09b0 \u09b0\u09c7\u099c\u0993\u09df\u09be\u09a8 \u09b6\u09c1\u09ad \u00a0 1.3 Python - Environment Setup Local Environment Setup Local Environment Setup \u098f\u09b0 \u0995\u09cd\u09b7\u09c7\u09a4\u09cd\u09b0\u09c7 \u0986\u09aa\u09a8\u09be\u09b0 \u0995\u09ae\u09cd\u09aa\u09bf\u0989\u099f\u09be\u09b0 \u098f\u09b0 \u09b8\u09be\u09b0\u09cd\u099a \u0987\u099e\u09cd\u099c\u09bf\u09a8\u09c7 \u099f\u09be\u0987\u09aa \u0995\u09b0\u09c1\u09a8 \u2018Python\u2019 \u098f\u09ac\u0982 \u09a6\u09c7\u0996\u09c1\u09a8 \u098f\u099f\u09bf \u0986\u09aa\u09a8\u09be\u09b0 \u0995\u09ae\u09cd\u09aa\u09bf\u0989\u099f\u09be\u09b0\u09c7 \u0987\u09a8\u09cd\u09b8\u099f\u09b2 \u0995\u09b0\u09be \u0986\u099b\u09c7 \u0995\u09bf\u09a8\u09be, \u098f\u09ac\u0982 \u09af\u09a6\u09bf \u09a5\u09be\u0995\u09c7 \u09a4\u09ac\u09c7 \u098f\u09b0 \u09ad\u09be\u09b0\u09cd\u09b8\u09a8\u099f\u09bf \u099c\u09c7\u09a8\u09c7 \u09b0\u09be\u0996\u09c1\u09a8\u0964 \u098f\u099b\u09dc\u09be\u0993 Python \u098f\u09b0 \u0985\u09ab\u09bf\u09b6\u09bf\u09df\u09be\u09b2 \u0993\u09df\u09c7\u09ac\u09b8\u09be\u0987\u099f (http:\/\/www.python.org\/) \u09a5\u09c7\u0995\u09c7\u2026","rel":"","context":"In "Python"","block_context":{"text":"Python","link":"http:\/\/bangla.sitestree.com\/?cat=1613"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":9953,"url":"http:\/\/bangla.sitestree.com\/?p=9953","url_meta":{"origin":24911,"position":2},"title":"\u09a8\u09a1.\u099c\u09c7\u098f\u09b8 \u098f\u09b0 \u09aa\u09b0\u09bf\u09ac\u09c7\u09b6 \u09a4\u09c8\u09b0\u09bf \u0995\u09b0\u09be (Node.js – Environment Setup)","author":"Author-Check- Article-or-Video","date":"July 25, 2015","format":false,"excerpt":"\u09b0\u09bf\u09a6\u0993\u09df\u09be\u09a8 \u09ac\u09bf\u09a8 \u09b6\u09be\u09ae\u09c0\u09ae \u00a0 \u0985\u09a8\u09b2\u09be\u0987\u09a8\u09c7 \u0985\u09aa\u09b6\u09a8 \u09aa\u09b0\u09c0\u0995\u09cd\u09b7\u09be \u0995\u09b0\u09be \u09a8\u09a1 \u099c\u09c7\u098f\u09b8 \u098f\u09b0 \u09aa\u09b0\u09bf\u09ac\u09c7\u09b6 \u09a8\u09a4\u09c1\u09a8 \u0995\u09b0\u09c7 \u09a4\u09c8\u09b0\u09bf \u0995\u09b0\u09be\u09b0 \u0995\u09bf\u099b\u09c1 \u09a8\u09c7\u0987 \u0995\u09be\u09b0\u09a3 \u0987\u09a4\u09cb\u09ae\u09a7\u09cd\u09af\u09c7\u0987 \u0985\u09a8\u09b2\u09be\u0987\u09a8\u09c7 \u09b8\u09c7\u0987 \u09aa\u09b0\u09bf\u09ac\u09c7\u09b6 \u09a4\u09c8\u09b0\u09bf \u09b9\u09df\u09c7 \u0986\u099b\u09c7, \u0985\u09a8\u09b2\u09be\u0987\u09a8\u09c7 \u09aa\u09cd\u09b0\u09be\u09aa\u09cd\u09af \u09b8\u0995\u09b2 \u0989\u09a6\u09be\u09b9\u09b0\u09a3 \u0986\u09ae\u09b0\u09be \u09a5\u09bf\u0993\u09b0\u09bf \u09a8\u09bf\u09df\u09c7 \u0995\u09be\u099c \u0995\u09b0\u09be\u09b0 \u09b8\u09ae\u09df\u09c7\u0987 \u09b8\u09ae\u09cd\u09aa\u09be\u09a6\u09a8 \u0995\u09b0\u09a4\u09c7 \u09aa\u09be\u09b0\u09ac\u0964 \u098f\u099f\u09bf \u0986\u09ae\u09be\u09a6\u09c7\u09b0 \u098f\u09ac\u09bf\u09b7\u09df\u09c7 \u09aa\u09dc\u09be\u09b6\u09c1\u09a8\u09be\u09b0 \u09b8\u09ae\u09df \u09ac\u09bf\u09ad\u09bf\u09a8\u09cd\u09a8 \u09ab\u09b2\u09be\u09ab\u09b2\u09b8\u09b9 \u09aa\u09b0\u09c0\u0995\u09cd\u09b7\u09be\u09a8\u09bf\u09b0\u09c0\u0995\u09cd\u09b7\u09be \u0995\u09b0\u09be\u09b0 \u0986\u09a4\u09cd\u09ae\u09ac\u09bf\u09b6\u09cd\u09ac\u09be\u09b8 \u09a6\u09c7\u09ac\u09c7\u0964\u2026","rel":"","context":"In "Root"","block_context":{"text":"Root","link":"http:\/\/bangla.sitestree.com\/?cat=1"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":14846,"url":"http:\/\/bangla.sitestree.com\/?p=14846","url_meta":{"origin":24911,"position":3},"title":"Learn some CentOS Linux : Redhat Linux","author":"Sayed","date":"June 29, 2019","format":false,"excerpt":"This is mostly about Centos Linux. However, this also shows commands related to how to install Magento 2 on LEMP (E for Nginx Web-server). Some PHP related stuff are there as well. LEMP + CentOS + Magento 2 : VM Image or one click install: I did not see readily\u2026","rel":"","context":"In "Magento 2"","block_context":{"text":"Magento 2","link":"http:\/\/bangla.sitestree.com\/?cat=1907"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11090,"url":"http:\/\/bangla.sitestree.com\/?p=11090","url_meta":{"origin":24911,"position":4},"title":"\u09aa\u09be\u0987\u09a5\u09a8 \u09aa\u09cd\u09b0\u09cb\u0997\u09cd\u09b0\u09be\u09ae\u09bf\u0982 : \u09b8\u09bf\u09a6\u09cd\u09a7\u09be\u09a8\u09cd\u09a4 \u0997\u09cd\u09b0\u09b9\u09a3 (Python Decision Making)","author":"","date":"October 24, 2015","format":false,"excerpt":"1.7 Python Decision Making Decision Making \u0985\u09b0\u09cd\u09a5\u09be\u09ce \u09b8\u09bf\u09a6\u09cd\u09a7\u09be\u09a8\u09cd\u09a4 \u09a8\u09c7\u09df\u09be\u099f\u09be \u09af\u09c7\u0995\u09cb\u09a8\u09cb \u09aa\u09cd\u09b0\u09cb\u0997\u09cd\u09b0\u09be\u09ae\u09bf\u0982 \u09b2\u09cd\u09af\u09be\u0999\u09cd\u0997\u09c1\u09df\u09c7\u099c \u098f\u09b0 \u098f\u0995\u099f\u09bf \u09aa\u09cd\u09b0\u099a\u09b2\u09bf\u09a4 \u0985\u0982\u09b6, \u09af\u09c7\u0996\u09be\u09a8\u09c7 \u09aa\u09cd\u09b0\u09cb\u0997\u09cd\u09b0\u09be\u09ae\u099f\u09bf \u0995\u09bf\u099b\u09c1 \u09aa\u09cd\u09b0\u09a6\u09a4\u09cd\u09a4 \u09b6\u09b0\u09cd\u09a4\u09c7\u09b0 \u0989\u09aa\u09b0 \u09ad\u09bf\u09a4\u09cd\u09a4\u09bf\u09a4\u09c7 \u0995\u09cb\u09a8 \u098f\u0995\u099f\u09bf \u09aa\u09cd\u09b0\u09cb\u09b8\u09c7\u09b8 \u09b8\u09a4\u09cd\u09af \u09a8\u09be\u0995\u09bf \u09ae\u09bf\u09a5\u09cd\u09af\u09be \u09b8\u09c7\u099f\u09be \u09af\u09be\u099a\u09be\u0987 \u0995\u09b0\u09c7\u0964 \u09aa\u09cd\u09b0\u09be\u09df \u09b8\u09ac \u09aa\u09cd\u09b0\u09cb\u0997\u09cd\u09b0\u09be\u09ae\u09bf\u0982 \u09b2\u09cd\u09af\u09be\u0982\u0997\u09c1\u09df\u09c7\u099c\u09c7\u0987 Decision Making \u098f\u09b0 \u09ac\u09cd\u09af\u09be\u09aa\u09be\u09b0 \u0986\u099b\u09c7\u0964 \u09a8\u09bf\u099a\u09c7\u09b0 \u099a\u09bf\u09a4\u09cd\u09b0\u09c7 Decision Making \u098f\u09b0 \u098f\u0995\u099f\u09be \u09ac\u09cd\u09af\u09be\u09b8\u09bf\u0995 \u09a7\u09be\u09b0\u09a3\u09be \u09a6\u09c7\u09df\u09be\u2026","rel":"","context":"In "Python"","block_context":{"text":"Python","link":"http:\/\/bangla.sitestree.com\/?cat=1613"},"img":{"alt_text":"Decision Making","src":"https:\/\/i0.wp.com\/bangla.salearningschool.com\/wp-content\/uploads\/2015\/10\/Decision-Making.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":25055,"url":"http:\/\/bangla.sitestree.com\/?p=25055","url_meta":{"origin":24911,"position":5},"title":"Learn and practice some CentOS Linux : Redhat Linux #Root","author":"Author-Check- Article-or-Video","date":"April 15, 2021","format":false,"excerpt":"This is mostly about Centos Linux. However, this also shows commands related to how to install Magento 2 on LEMP (E for Nginx Web-server). Some PHP related stuff are there as well. Let's Encrypt included. LEMP + CentOS + Magento 2 : VM Image or one click install: I did\u2026","rel":"","context":"In "FromSitesTree.com"","block_context":{"text":"FromSitesTree.com","link":"http:\/\/bangla.sitestree.com\/?cat=1917"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=\/wp\/v2\/posts\/24911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=24911"}],"version-history":[{"count":0,"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=\/wp\/v2\/posts\/24911\/revisions"}],"wp:attachment":[{"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=24911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=24911"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/bangla.sitestree.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=24911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}