A Data Warehouse is a storage of an organization’s historical data. Usually, data warehouse is built to provide a decision support system to the management. Data warehouse can be mined (using data mining) and queried to collect overall and summarized business information that significantly helps in decision making. It can provide trends over many years.

Operational databases are optimized for day to day operation access. Here, the relationships are normalized and further divided into more relations/tables to make accessing faster that also reduces data traffic among the systems. But a data warehouse does the reverse, where relationships/tables are de-normalized as much as possible to provide the required information to the management.

As a data warehouse provides summarized information de-normalization helps, otherwise the data would be required to collect from many tables and the queries may be running over data for 10 years (millions/billions of records). Usually, data warehouse is created separately from the operational databases.

Microsoft SQL Server and Oracle both have supports and tools to create Data warehouse Systems. There are also free tools for the purpose even for My SQL. From: http://sitestree.com/?p=3569
Categories:Data Warehouse, Data Warehouse – 001, Data Warehouse, Data Warehouse Misc
Tags:
Post Data:2016-07-05 15:37:20

Shop Online: https://www.ShopForSoul.com/
(Big Data, Cloud, Security, Machine Learning): Courses: http://Training.SitesTree.com
In Bengali: http://Bangla.SaLearningSchool.com
http://SitesTree.com
8112223 Canada Inc./JustEtc: http://JustEtc.net (Software/Web/Mobile/Big-Data/Machine Learning)
Shop Online: https://www.ShopForSoul.com/
Medium: https://medium.com/@SayedAhmedCanada

• Real internet money/business lies in B2B not in B2C.
• Decision Support System (DSS):
  • Full spectrum of systems that allow/help management to take decisions, such as, reporting, OLAP, data mining.
• Java provides support for web enabled DSS
• JDM is Java’s Data mining/BI API.
• BI Applications: Balanced Score-card, Activity Based Costing.
• Data warehouse elements
  • Operational Environment: Maintains day to day activities. Raw Data
  • Independent data mart: A small data warehouse specific to one topic/area/subject
  • Extraction: Extraction of data from operational systems
  • Extraction Store: Data storage after extraction
  • Transformation and cleansing: Before putting data into the central/main warehouse data needs to be cleaned up
  • Extraction log: Create a log in the transformation and cleansing process
  • External source: Data originated outside the organization is also incorporated into the data warehouse.
  • Data administrator: The person who ensures the quality of data
  • Central Repository: Stores all data and meta-data
  • Meta data: Data about data. Example: the extraction log.
  • Dependent Data Mart: Relies on the data warehouse for the sources of its data
• Distributed Data Warehouse: Warehouse data is physically distributed across a company’s information infrastructure.
• Data mining types: classification, and estimation.
  • Classification: categorize customers,
  • Estimation: predict or estimate some numerical value
• Decision trees: modeling technique helps in deciding. Different alternatives are drawn as paths where nodes represent the selected alternative. edges/paths may carry some probability values as selection/influencing criteria.
• Neural networks: Try to mimic the work nature of human brain to take a decision.
• General modeling: Well-suited for categorizing.
• Balanced Scorecard application: views the health and well-being of the organization by more than its financial data. We need to define: Vision, Strategy, Strategic themes, Strategic Objectives, Key performance indicators.
• Activity based costing application:
• Three types of organizations:
  • Engineer driven: They know how to make a product. Than sales it through sales force.
  • Sales Driven: They know how to sale a particular product, then make the product and sell it.
  • Customer driven: They Know what customers want, then make the products and sale them. Customer driven organizations are the most successful.

From: http://sitestree.com/?p=3567
Categories:Data Warehouse, Data Warehouse – 001, Data Warehouse, Data Warehouse Misc
Tags:
Post Data:2016-07-04 15:22:58

Shop Online: https://www.ShopForSoul.com/
(Big Data, Cloud, Security, Machine Learning): Courses: http://Training.SitesTree.com
In Bengali: http://Bangla.SaLearningSchool.com
http://SitesTree.com
8112223 Canada Inc./JustEtc: http://JustEtc.net (Software/Web/Mobile/Big-Data/Machine Learning)
Shop Online: https://www.ShopForSoul.com/
Medium: https://medium.com/@SayedAhmedCanada

• Most BI projects usually fail. It is not due to the errors in the construction steps but due to the inappropriate design steps and methodologies.
• Waterfall method is not appropriate for BI. A method like developmental spiral may be more appropriate. Ref: [W. A. Giovinazzo ]
• Steps in developmental spiral: Definition, Analysis, Design, Development, Implementation/Deployment, Maintenance, Evaluation
• Developmental spiral method is effective for BI only when Object Oriented approach is taken
• In developmental spiral, object modeling is used to represent objects in the related business areas. In object modeling, objects are represented as rectangles, the rectangles can be treated as classes. Classes/objects can have sub-classes/sub-objects. Objects have attributes. Objects also have some special properties like: accelerators, stops.
• Definition – first step of developmental spiral: Define a short description what you want to build – like the problem statement in a research proposal. Example: I want to build a data mart that will provide data to support the analysis of customer demographics such as age, gender, and income for all product lines
• Analysis Phase: In this step, analyze the requirements of the BI system, the nature of the BI system, the expectations from the BI system. Interviewing different parties will help a lot:
• Interview the sponsor to get a detail understanding of the mission. The questions will include: Data-related(which objects are related to the business needs?), user-related: who are the users, system evaluation: what parameters will make the project a success?
• Interviewing management: To understand management’s expectation of the data warehouse. Questions may include: User profile related (what types of systems you use/will use to access BI?), System vision: what do you see as the objective of the data warehouse, System Evaluation: what parameters will make the project a success?
• Interviewing Users : Interviewing users who will use the system
• Interviewing information technology staffs: To know the current state of the organization’s information systems
• Design phase: output – object model, implementation model
• Start with the analysis phase information. The design phase is complete when all the objects related to the mission are well defined. An object diagram needs to be drawn showing all objects, relationships, attributes.
• Another chart listing all the attributes and their data types will be useful. Objects will have relations like super classes or sub-classes.
• Cardinality relationships: exactly one, one to many, zero or one, zero to many
• aggregation may be required in some cases. Aggregation: an object is composed of other objects.
• Implementation Model: Data warehouse databases are multi dimensional databases. Objects are represented in more than two dimensions. Common practice is: three dimensional databases. Such as an object/table represented with three dimensions (called cube) like: product, dealer, and time. RDBMS’s are two dimensional.
• Data cubes: have six possible different views [permutation of three dimensions].
• Why not always multidimensional databases: because of space requirements.
• Star schema: Provides a multidimensional flavor in two dimensional relational databases.
• Star schema uses a concept called fact tables to bind dimensions to create a multidimensional space.
• Denormalization of the tables are utilized in star schema to create multi dimension.
• Dimension table: Think about a three dimensional cube. Each wall represents a dimension table. Or think it as a mathematical combination of the dimensions.
• All objects represented in the dimensions can slowly change over time – slowly changing dimensions. There are many approaches to address this issue.
• Snowflakes: normalizing dimensions – not always a great thing
• Implementation considerations: Parallel processing, Bitmapped Indexing, Star Query Optimization, Summation Tables

  From: http://sitestree.com/?p=3565
Categories:Data Warehouse, Data Warehouse – 001, Data Warehouse, Data Warehouse Misc
Tags:
Post Data:2016-07-06 15:54:10

Shop Online: https://www.ShopForSoul.com/
(Big Data, Cloud, Security, Machine Learning): Courses: http://Training.SitesTree.com
In Bengali: http://Bangla.SaLearningSchool.com
http://SitesTree.com
8112223 Canada Inc./JustEtc: http://JustEtc.net (Software/Web/Mobile/Big-Data/Machine Learning)
Shop Online: https://www.ShopForSoul.com/
Medium: https://medium.com/@SayedAhmedCanada

The idea was :

use two servers; server1, and server2
create user: user1 in both server1 and server2
use nmtui to assign IP Addresses to both
use ifdown and ifup to check if IPs are assigned properly
change /etc/hosts and add the server ips and names
use ping to know that you can reach each other
use ssh-key gen to create key files
use ssh-copy-id to transfer public key file to remote hosts
then use ssh commands to reach to without passwords
try scp and sftp : to copy files/folder to and from

Then use TCP wrappers to allow or deny remote hosts, users, or domains

SSH:

2 useradd user1
3 passwd user1
4 ip addr
5 nmtui
6 vim /etc/hosts
7 ping server2
8 ip addr
9 nmtui
10 ifdown ens33
11 ifup ens33
12 ping 192.168.1.200
13 vim /etc/hosts
14 ping server2
15 su user1

ssh-keygen
1 ssh server2 /bin/ls -l
2 scp file1 server2:
3 touch file1
4 scp file1 server2:
5 scp server2:/usr/bin/zip .

7 sftp server2
8 su root

10 w
11 last

13 tail /var/log/secure

15 sudo tail /var/log/secure

19 getcwd

24 ssh-keygen

27 ssh-copy-id -i ~/.ssh/id_rsa.pub server2

31 vim .ssh/authorized_keys
32 ssh server2

———————

TCP wrappers
/etc/hosts.allow : to allow to services from whom
/etc/hosts.allow : to deny to services from whom

Install TCP Wrappers. I used rpm as I turned off internet in the VMware workstation I used

27 rpm -ivh tcp_wrappers*
28 yum info tcp_wrappers
29 ls /etc/hosts.allow
30 vim /etc/hosts.allow
30 vim /etc/hosts.allow

example rules on hosts.allow

All:All
All:user1
All:user1@server2
All:.justetc.com
sshd:LOCAL
vsftpd:192.168.1.200
All Except sshd:192.168.1.200 From: http://sitestree.com/?p=11978
Categories:DevOps
Tags:
Post Data:2017-12-15 12:37:07

Shop Online: https://www.ShopForSoul.com/
(Big Data, Cloud, Security, Machine Learning): Courses: http://Training.SitesTree.com
In Bengali: http://Bangla.SaLearningSchool.com
http://SitesTree.com
8112223 Canada Inc./JustEtc: http://JustEtc.net (Software/Web/Mobile/Big-Data/Machine Learning)
Shop Online: https://www.ShopForSoul.com/
Medium: https://medium.com/@SayedAhmedCanada