{"id":17002,"date":"2020-04-19T06:33:29","date_gmt":"2020-04-19T10:33:29","guid":{"rendered":"http:\/\/bangla.salearningschool.com\/recent-posts\/python-ecommerce-part-1-merge-multiple-supplier-data-files-into-one-file\/"},"modified":"2020-04-19T15:16:54","modified_gmt":"2020-04-19T19:16:54","slug":"python-ecommerce-part-1-merge-multiple-supplier-data-files-into-one-file","status":"publish","type":"post","link":"http:\/\/bangla.sitestree.com\/?p=17002","title":{"rendered":"Python: Ecommerce: Part \u2014 1: Merge Multiple Supplier Data Files into One File"},"content":{"rendered":"
\n
\n
\n
\n
\n

Section: Merge multiple Supplier Data Files<\/h1>\n

All code in one block<\/strong><\/p>\n

#!\/usr\/bin\/env python\n# coding: utf-8<\/span># # Section: Merge multiple Supplier Data Files\n#<\/span># In[1]:<\/span># if there is a need to merge multiple files \u2014 use this block\nimport os;\nimport glob;\nimport pandas as pd;<\/span># supplier data files\/feeds are kept here\ndata_folder = \u2018data-supplier-2019\u201304\u201314\/supplier-raw-data\/\u2019;\nos.chdir(data_folder);<\/span># In[6]:<\/span># show all data feed file name\n# file extension for supplier data file\nextension = \u2018csv\u2019;\nall_filenames = [i for i in glob.glob(\u2018*.{}\u2019.format(extension))]\nall_filenames<\/span># In[7]:<\/span># total number of rows combined all data files\/feeds\nrow_total_count = 0\nfor f in all_filenames:\n df_s = pd.read_csv(f)\n print(df_s.shape, f)\n row_total_count += df_s.shape[0]\nrow_total_count # print(row_total_count)<\/span># In[8]:<\/span># combine all files in the list\ncombined_csv = pd.concat([pd.read_csv(f) for f in all_filenames]);\ncombined_csv.shape<\/span># In[10]:<\/span># export combined data to a csv file\ncombined_csv.to_csv( \u201c..\/all_supplier_products_2019_04_14.csv\u201d, index=False, encoding=\u2019utf-8-sig\u2019)<\/span># In[13]:<\/span># read csv data file and show data on the screen\ndf = pd.read_csv(\u2018..\/all_supplier_products_2019_04_14.csv\u2019);\ndf.head()<\/span><\/pre>\n
The following is from Jupyter Notebook: Cell By Cell Display. Output data are also shown<\/strong><\/p>\n
In [1]:<\/p>\n
# if there is a need to merge multiple files -- use this block<\/em><\/span>import<\/strong> os;<\/span>import<\/strong> glob;<\/span>import<\/strong> pandas as<\/strong> pd;<\/span># supplier data files\/feeds are kept here<\/em><\/span>data_folder =<\/strong> 'data-supplier-2019-04-14\/supplier-raw-data\/';<\/span>os.chdir(data_folder);<\/span><\/pre>\n
In [6]:<\/p>\n
# show all data feed file name<\/em><\/span># file extension for supplier data file<\/em><\/span>extension =<\/strong> 'csv';<\/span>all_filenames =<\/strong> [i for<\/strong> i in<\/strong> glob.glob('*.{}'.format(extension))]<\/span>all_filenames<\/span><\/pre>\n
Out[6]:<\/p>\n
['data_feeds_5e95c25a1f7f6.csv',\n 'data_feeds_5e95c2962d471.csv',\n 'data_feeds_5e95c2d255409.csv',\n 'data_feeds_5e95c30e63423.csv',\n 'data_feeds_5e95c38646478.csv',\n 'data_feeds_5e95c5dd76370.csv']<\/span><\/pre>\n
In [7]:<\/p>\n
# total number of rows combined all data files\/feeds<\/em><\/span>row_total_count =<\/strong> 0<\/span>for<\/strong> f in<\/strong> all_filenames:<\/span>df_s =<\/strong> pd.read_csv(f)<\/span>print(df_s.shape, f)<\/span>row_total_count +=<\/strong> df_s.shape[0]<\/span>row_total_count # print(row_total_count)<\/em><\/span>(8058, 40) data_feeds_5e95c25a1f7f6.csv\n(7, 40) data_feeds_5e95c2962d471.csv\n(1, 40) data_feeds_5e95c2d255409.csv<\/span>... ....\n(1072, 40) data_feeds_5e95c565d6e30.csv\n(4833, 40) data_feeds_5e95c5dd76370.csv<\/span><\/pre>\n
Out[7]:<\/p>\n
55690<\/span><\/pre>\n
In [8]:<\/p>\n
# combine all files in the list<\/em><\/span>combined_csv =<\/strong> pd.concat([pd.read_csv(f) for<\/strong> f in<\/strong> all_filenames]);<\/span>combined_csv.shape<\/span><\/pre>\n
Out[8]:<\/p>\n
(55690, 40)<\/span><\/pre>\n
In [10]:<\/p>\n
# export combined data to a csv file<\/em><\/span>combined_csv.to_csv( \"..\/all_supplier_products_2019_04_14.csv\", index=False<\/strong>, encoding=<\/strong>'utf-8-sig')<\/span><\/pre>\n
In [13]:<\/p>\n
df =<\/strong> pd.read_csv('..\/all_supplier_products_2019_04_14.csv');<\/span>df.head()<\/span><\/pre>\n
Out[13]:<\/p>\n
Product ID Model Code Full Product NameShort Product NameProduct URLCategory NameCategory URLSubcategory NameSubcategory URLDate Product Was Launched\u2026Related ProductsRelated AccessoriesWeight KgHeight mmWidth mmDepth mmVideo linkRetail PriceStock statusDate Back0107890POU_0850GV7YPull Rope Fitness Exercises Resistance Bands L\u2026Pull Rope Fitness<\/p>\n
***. ***. ***<\/em><\/strong><\/p>\n
Note: Older short-notes from this site are posted on Medium:\u00a0<\/em><\/strong>https:\/\/medium.com\/@SayedAhmedCanada<\/a><\/p>\n
*** . *** *** . *** . *** . ***<\/p>\n
Sayed Ahmed<\/em><\/strong><\/p>\n
BSc. Eng. in Comp. Sc. & Eng. (BUET)<\/em><\/strong><\/p>\n
MSc. in Comp. Sc. (U of Manitoba, Canada)<\/em><\/strong><\/p>\n
MSc. in Data Science and Analytics (Ryerson University, Canada)<\/em><\/strong><\/p>\n
Linkedin<\/em><\/strong>:\u00a0<\/em>https:\/\/ca.linkedin.com\/in\/sayedjustetc<\/em><\/a><\/p>\n
Blog<\/em><\/strong>:\u00a0<\/em>http:\/\/Bangla.SaLearningSchool.com<\/em><\/a>,\u00a0<\/em>http:\/\/SitesTree.com<\/em><\/a><\/p>\n
Training Courses:\u00a0<\/em><\/strong>http:\/\/Training.SitesTree.com<\/em><\/a><\/p>\n
8112223 Canada Inc\/Justetc<\/em><\/strong>:\u00a0<\/em>http:\/\/JustEtc.net<\/em><\/a><\/p>\n
Facebook Groups\/Forums to discuss (Q & A):<\/em><\/strong><\/p>\n
https:\/\/www.facebook.com\/banglasalearningschool<\/a><\/p>\n
https:\/\/www.facebook.com\/justetcsocial<\/a><\/p>\n
Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others.\u00a0<\/em>http:\/\/sitestree.com\/training\/<\/a><\/p>\n<\/div>\n<\/div>\n<\/section>\n<\/div>\n<\/article>\n
\n
\n
\n
\n
\n
\n
Build Ecommerce Software and Systems<\/h2>\n
\n
Build Ecommerce Software and Systems<\/h4>\n<\/div>\n