{"id":17004,"date":"2020-04-19T06:37:36","date_gmt":"2020-04-19T10:37:36","guid":{"rendered":"http:\/\/bangla.salearningschool.com\/recent-posts\/python-ecommerce-part-3-remove-unwanted-category-and-products-also-remove-products-based-on-words-in-the-title-after-merging-all-supplier-data-files-into-one-file\/"},"modified":"2020-04-19T15:14:56","modified_gmt":"2020-04-19T19:14:56","slug":"python-ecommerce-part-3-remove-unwanted-category-and-products-also-remove-products-based-on-words-in-the-title-after-merging-all-supplier-data-files-into-one-file","status":"publish","type":"post","link":"http:\/\/bangla.sitestree.com\/?p=17004","title":{"rendered":"Python: Ecommerce: Part \u2014 3: Remove Unwanted Category (and Products), Also, remove products based on Words in the Title \u2014 after Merging All Supplier Data Files into One File"},"content":{"rendered":"
\n
\n
\n
\n

You could as well remove products that are not allowed in a country or in a market place as well as products that you are not authorized to sell (some brands)<\/p>\n

All Code in One Block. Please check the other parts of this series\/publication<\/p>\n

The code could be simplified\/reduced. You could join multiple blocks into one just by keeping the words\/category names in a list; and then filtering against that list. You could as well join conditions using and (&) or or-operations (|) to reduce the number of lines of code.<\/p>\n

# # Section Remove products that have slang words<\/span># In[24]:<\/span>unique_sorted_data[\u2018Category Name\u2019].unique()<\/span># In[30]:<\/span># Remove products from a category that you do not want to sell<\/span># Apparel<\/span>unique_sorted_data_filter_category = unique_sorted_data [\n ~( unique_sorted_data[\u2018Category Name\u2019].str.contains(\u201cApparel\u201d, case = False, na=False ) )\n ];<\/span># android TV Box<\/span>unique_sorted_data_filter_category = unique_sorted_data_filter_category [\n ~( unique_sorted_data_filter_category[\u2018Category Name\u2019].str.contains(\u201cTV Box\u201d, case = False, na=False ) )\n ];<\/span># Laser Products\nunique_sorted_data_filter_category = unique_sorted_data_filter_category [\n ~( unique_sorted_data_filter_category[\u2018Category Name\u2019].str.contains(\u201cLaser\u201d, case = False, na=False ) )\n ];<\/span># Costume\nunique_sorted_data_filter_category = unique_sorted_data_filter_category [\n ~( unique_sorted_data_filter_category[\u2018Category Name\u2019].str.contains(\u201cCostume\u201d, case = False, na=False ) )\n ];<\/span># Showing the count after removal\nunique_sorted_data_filter_category[\u2018Category Name\u2019].unique(), unique_sorted_data_filter_category.shape<\/span># In[31]:<\/span>#################<\/span># section Remove products that have slang\/bad words in the name -- you do not need this block - was for testing only. Another block will do this job<\/span>unique_sorted_data_filter_1 = unique_sorted_data[ \n ( unique_sorted_data[\u2018Full Product Name\u2019].str.contains(\u201cSlang 1\u201c, case = False, na=False) )\n \n | ( unique_sorted_data[\u2018Full Product Name\u2019].str.contains(\u201cSlang 2\u201c, case = False, na=False) )\n \n | ( unique_sorted_data[\u2018Full Product Name\u2019].str.contains(\u201cSlang 3\u201c, case = False, na=False) )\n \n | ( unique_sorted_data[\u2018Full Product Name\u2019].str.contains(\u201cSlang 4\u201c, case = False, na=False) )<\/span>] #[ {\u2018Full Product Name\u2019, \u2018Category Name\u2019}];\nunique_sorted_data_filter_1.shape #, unique_sorted_data_filter_1.head(1) #, \u201c\\n \u201c, unique_sorted_data_filter_1.shape<\/span>########################<\/span># In[38]:<\/span># Remove products that have slang words in the product name<\/span>unique_sorted_filtered_data = unique_sorted_data_filter_category [\n ~( unique_sorted_data_filter_category[\u2018Full Product Name\u2019].str.contains(\u201cBad word 1\u201c, case = False, na=False ) ) ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cBad word 2\u201c, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cBad word 3\u201c, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cBad word 4\u201d, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cBad word 5\u201c, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cBad word 1\u201c, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cTracker\u201d, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cLaser\u201d, case = False, na=False ) )\n ];<\/span># brands that you do not want to sell\nprint(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cVKworld\u201d, case = False, na=False ) )\n ];<\/span>\nprint(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cSamsung\u201d, case = False, na=False ) )\n ];<\/span># video streaming TV Box HDMI -- products are sensitive (intellectual rights)\nprint(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cCar Video\u201d, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cStreaming\u201d, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data = unique_sorted_filtered_data [\n ~( unique_sorted_filtered_data[\u2018Full Product Name\u2019].str.contains(\u201cHDMI\u201d, case = False, na=False ) )\n ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data.shape<\/span># In[39]:<\/span>unique_sorted_filtered_data.to_csv(\u201c..\/all_supplier_data_unique_sorted_and_filtered.csv\u201d);<\/span><\/pre>\n
From Jupyter Notebook: Cell by Cell with Output<\/strong><\/h1>\n
Section Remove products that have slang\/bad words or similar. Or Brands\/Products that you do not want to sell (not allowed to sell\/resell\/restricted products or similar): I know the grammar is not right. Sorry, not fixing it.<\/h1>\n
In [24]:<\/p>\n
unique_sorted_data['Category Name'].unique()<\/span><\/pre>\n
Out[24]:<\/p>\n
array(['Toys & Games', 'Drone & Quadcopter', 'Cool Gadgets',\n       'Office supplies', 'Novelty Costumes & Accessories',\n       \"Women's Jewelry\", 'External Parts', 'Vehicle Electronics Devices',\n       'Replacement Parts', 'Internal Parts', 'Lamps and Accessories',\n       'Video Games', 'Hair Care', 'Skin Care',\n       'Makeup Tool & Accessories', 'Household Products',\n       \"Women's Accessories\", \"Women's Apparel\", 'Home accessories',\n       'Oral Respiratory Protection', \"Men's Accessories\",\n       \"Men's Apparel\", \"Girl's Apparel\", 'Cell Phone Accessories',\n       'Health Care', 'Electronic Accessories', 'Health tools',\n       'Computer Peripherals', 'Audio & Video Gadgets', nan,\n       'Headrest Monitors & DVD Players', 'Car DVR',\n       'Camera Equipment \/ Accessories', 'Personal Care',\n       'Laser Gadgets & Measuring Tools', 'Accessories',\n       'Electronic Cigarettes', 'Sports Action Camera',\n       'Android TV Box \/ Stick', 'Sports & Body Building',\n       'Smart Watches', 'Security & Surveillance', 'Android Tablets',\n       'Musical Instruments & Accessorie', 'LED', 'Outdoor Recrections',\n       'Tools & Home Decor', 'Home, Kitchen & Garden', 'Home Electrical',\n       'Bedding & Bath', 'Camping & Hiking', 'Drives & Storage',\n       'Pet Supplies', 'Hunting & Fishing', 'Garden & Lawn',\n       'Medical treatments', 'Android Smartphones', 'Car Video',\n       'Cell Phones', 'Cycling', 'Solar Products', 'Doogee Phones',\n       'Rugged Phones', 'Ulefone Phones', 'Xiaomi Phones', 'Huawei Phone',\n       'Lenovo Phones', 'Refurbished iPhones', 'Samsung Phones',\n       'Water Sport', 'Tools & Equipment', 'Repair Accessories',\n       'Body protection', 'Disinfection and sterilization', \"Men's Care\",\n       'Cleaning Supplies', 'Baby Girls Apparel', \"Women's Bags\",\n       \"Women's Shoes\", \"Men's Jewelry\", 'Baby Boys Apparel',\n       \"Boy's Apparel\", \"Girl's Shoes\", \"Girl's Jewelry\", \"Boy's Shoes\",\n       'kN95\/KF94 Mask', 'Flash Drives + Memory Cards',\n       '6-7 Inch Android Phones', 'Apple Phones', 'Xiaomi Phone',\n       'Laptops & Tablets', 'Apple iPad', 'Musical Instruments',\n       'Computer Accessories', 'Ball Games', \"Boy's Jewelry\"],\n      dtype=object)<\/span><\/pre>\n
In [30]:<\/p>\n
# Remove products from a category that you do not want to sell<\/em><\/span>unique_sorted_data_filter_category =<\/strong> unique_sorted_data [ ~<\/strong>( unique_sorted_data['Category Name'].str.contains(\"Apparel\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>\u200bunique_sorted_data_filter_category = <\/strong>unique_sorted_data_filter_category [ ~<\/strong>( unique_sorted_data_filter_category['Category Name'].str.contains(\"TV Box\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>unique_sorted_data_filter_category = <\/strong>unique_sorted_data_filter_category [~<\/strong>( unique_sorted_data_filter_category['Category Name'].str.contains(\"Laser\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>unique_sorted_data_filter_category =<\/strong> unique_sorted_data_filter_category [ ~<\/strong>( unique_sorted_data_filter_category['Category Name'].str.contains(\"Costume\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )<\/span>];<\/span>unique_sorted_data_filter_category['Category Name'].unique(), unique_sorted_data_filter_category.shape<\/span><\/pre>\n
Out[30]:<\/p>\n
(array(['Toys & Games', 'Drone & Quadcopter', 'Cool Gadgets',\n        'Office supplies', \"Women's Jewelry\", 'External Parts',\n        'Vehicle Electronics Devices', 'Replacement Parts',\n        'Internal Parts', 'Lamps and Accessories', 'Video Games',\n        'Hair Care', 'Skin Care', 'Makeup Tool & Accessories',\n        'Household Products', \"Women's Accessories\", 'Home accessories',\n        'Oral Respiratory Protection', \"Men's Accessories\",\n        'Cell Phone Accessories', 'Health Care', 'Electronic Accessories',\n        'Health tools', 'Computer Peripherals', 'Audio & Video Gadgets',\n        nan, 'Headrest Monitors & DVD Players', 'Car DVR',\n        'Camera Equipment \/ Accessories', 'Personal Care', 'Accessories',\n        'Electronic Cigarettes', 'Sports Action Camera',\n        'Sports & Body Building', 'Smart Watches',\n        'Security & Surveillance', 'Android Tablets',\n        'Musical Instruments & Accessorie', 'LED', 'Outdoor Recrections',\n        'Tools & Home Decor', 'Home, Kitchen & Garden', 'Home Electrical',\n        'Bedding & Bath', 'Camping & Hiking', 'Drives & Storage',\n        'Pet Supplies', 'Hunting & Fishing', 'Garden & Lawn',\n        'Medical treatments', 'Android Smartphones', 'Car Video',\n        'Cell Phones', 'Cycling', 'Solar Products', 'Doogee Phones',\n        'Rugged Phones', 'Ulefone Phones', 'Xiaomi Phones', 'Huawei Phone',\n        'Lenovo Phones', 'Refurbished iPhones', 'Samsung Phones',\n        'Water Sport', 'Tools & Equipment', 'Repair Accessories',\n        'Body protection', 'Disinfection and sterilization', \"Men's Care\",\n        'Cleaning Supplies', \"Women's Bags\", \"Women's Shoes\",\n        \"Men's Jewelry\", \"Girl's Shoes\", \"Girl's Jewelry\", \"Boy's Shoes\",\n        'kN95\/KF94 Mask', 'Flash Drives + Memory Cards',\n        '6-7 Inch Android Phones', 'Apple Phones', 'Xiaomi Phone',\n        'Laptops & Tablets', 'Apple iPad', 'Musical Instruments',\n        'Computer Accessories', 'Ball Games', \"Boy's Jewelry\"],\n       dtype=object), (47826, 40))<\/span><\/pre>\n
In [31]:<\/p>\n
# sect<\/em><\/span><\/pre>\n
In [38]:<\/p>\n
# Remove products that have slang words in the product name<\/em><\/span>unique_sorted_filtered_data =<\/strong> unique_sorted_data_filter_category [ ~<\/strong>( unique_sorted_data_filter_category['Full Product Name'].str.contains(\"Bad word 1\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span>unique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Bad word 2\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Bad word 3\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span># product\nprint(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Tracker\", case =<\/strong> False<\/strong>, na=False<\/strong> ) ) ];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Laser\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span># remove brands\nunique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"VKworld\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span>unique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Samsung\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span># video, streaming, HDMI\nunique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Car Video\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span>unique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [ ~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"Streaming\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);<\/span>unique_sorted_filtered_data =<\/strong> unique_sorted_filtered_data [~<\/strong>( unique_sorted_filtered_data['Full Product Name'].str.contains(\"HDMI\", case =<\/strong> False<\/strong>, na=False<\/strong> ) )];<\/span>print(unique_sorted_filtered_data.shape);\nunique_sorted_filtered_data.shape<\/span>(47001, 40)\n(46988, 40)\n(46968, 40)\n(46968, 40)\n(46959, 40)\n(46959, 40)\n(46441, 40)\n(46388, 40)\n(46381, 40)\n(45357, 40)\n(45349, 40)\n(45338, 40)\n(44911, 40)<\/span><\/pre>\n
Out[38]:<\/p>\n
(44911, 40)<\/span><\/pre>\n
In [39]:<\/p>\n
# send the filtered data to a file\nunique_sorted_filtered_data.to_csv(\"..\/all_supplier_data_unique_sorted_and_filtered.csv\");<\/span><\/pre>\n
***. ***. ***<\/em><\/strong><\/p>\n
Note: Older short-notes from this site are posted on Medium:\u00a0<\/em><\/strong>https:\/\/medium.com\/@SayedAhmedCanada<\/a><\/p>\n
*** . *** *** . *** . *** . ***<\/p>\n
Sayed Ahmed<\/em><\/strong><\/p>\n
BSc. Eng. in Comp. Sc. & Eng. (BUET)<\/em><\/strong><\/p>\n
MSc. in Comp. Sc. (U of Manitoba, Canada)<\/em><\/strong><\/p>\n
MSc. in Data Science and Analytics (Ryerson University, Canada)<\/em><\/strong><\/p>\n
Linkedin<\/em><\/strong>:\u00a0<\/em>https:\/\/ca.linkedin.com\/in\/sayedjustetc<\/em><\/a><\/p>\n
Blog<\/em><\/strong>:\u00a0<\/em>http:\/\/Bangla.SaLearningSchool.com<\/em><\/a>,\u00a0<\/em>http:\/\/SitesTree.com<\/em><\/a><\/p>\n
Training Courses:\u00a0<\/em><\/strong>http:\/\/Training.SitesTree.com<\/em><\/a><\/p>\n
8112223 Canada Inc\/Justetc<\/em><\/strong>:\u00a0<\/em>http:\/\/JustEtc.net<\/em><\/a><\/p>\n
Facebook Groups\/Forums to discuss (Q & A):<\/em><\/strong><\/p>\n
https:\/\/www.facebook.com\/banglasalearningschool<\/a><\/p>\n
https:\/\/www.facebook.com\/justetcsocial<\/a><\/p>\n
Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others.\u00a0<\/em>http:\/\/sitestree.com\/training\/<\/a><\/p>\n<\/div>\n<\/div>\n<\/section>\n<\/article>\n
\n
\n
\n
\n
\n
\n
Build Ecommerce Software and Systems<\/h2>\n
\n
Build Ecommerce Software and Systems<\/h4>\n<\/div>\n