Bayesian Statistics and Machine Learning

“Bayesian inference is a method of statistical inference in which Bayes‘ theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and especially in mathematical statistics.”

en.wikipedia.org › wiki › Bayesian_inference

Bayesian inference – Wikipedia

“Firstly, (statistical) inference is the process of deducing properties about a population or probability distribution from data”

“Bayesian inference is therefore just the process of deducing properties about a population or probability distribution from data using Bayes’ theorem. That’s it.”
https://towardsdatascience.com/probability-concepts-explained-bayesian-inference-for-parameter-estimation-90e8930e5348

Introduction to Bayesian Inference

https://blogs.oracle.com/datascience/introduction-to-bayesian-inference

”

Bayesian Linear Regression

In the Bayesian viewpoint, we formulate linear regression using probability distributions rather than point estimates. The response, y, is not estimated as a single value, but is assumed to be drawn from a probability distribution. The model for Bayesian Linear Regression with the response sampled from a normal distribution is:”

https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7

“Bayesian model selection

Tom Minka

Bayesian model selection uses the rules of probability theory to select among different hypotheses. It is completely analogous to Bayesian classification.”

”

”

http://alumni.media.mit.edu/~tpminka/statlearn/demo/

Logistic Regression from Bayes’ Theorem

https://www.countbayesie.com/blog/2019/6/12/logistic-regression-from-bayes-theorem

Kernel Trick and Kernels
https://svivek.com/teaching/lectures/slides/svm/kernels.pdf

“When talking about kernels in machine learning, most likely the first thing that comes into your mind is the support vector machines (SVM)”

https://medium.com/@zxr.nju/what-is-the-kernel-trick-why-is-it-important-98a98db0961d

Gaussian Processes

“or why I don’t use SVMs”

https://mlss2011.comp.nus.edu.sg/uploads/Site/lect1gp.pdf

Gaussian Process Classification and Active Learning with Multiple Annotators

http://proceedings.mlr.press/v32/rodrigues14.pdf

“Assumed density filtering is an online inference algorithm. that incrementally updates the posterior over W after ob- serving new evidence.”

https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/12391/11777

“Expectation propagation (EP) is a technique in Bayesian machine learning. EP finds approximations to a probability distribution. It uses an iterative approach that leverages the factorization structure of the target distribution.

en.wikipedia.org › wiki › Expectation_propagation

Expectation propagation – Wikipedia

”

rejection sampling

”

In numerical analysis and computational statistics, rejection sampling is a basic technique used to generate observations from a distribution. It is also commonly called the acceptance-rejection method or “accept-reject algorithm” and is a type of exact simulation method. The method works for any distribution in with a density.

https://en.wikipedia.org/wiki/Rejection_sampling

”

Bayesian Deep learning

“However, graphical networks such as Bayesian networks are useful to model uncertainty along with causal inference and logic deduction. In this regard, Bayesian Deep learning combines perception (deep learning) with strong probabilistic inference which can estimate uncertainty.

www.quora.com › How-different-is-Bayesian-deep-learning-from-deep-…

How different is Bayesian deep learning from deep learning? – Quora

”

*** . . *** . ***
Note: Older short-notes from this site are posted on Medium: https://medium.com/@SayedAhmedCanada

*** . *** *** . *** . *** . ***

Sayed Ahmed

BSc. Eng. in Comp. Sc. & Eng. (BUET)
MSc. in Comp. Sc. (U of Manitoba, Canada)
MSc. in Data Science and Analytics (Ryerson University, Canada)
Linkedin: https://ca.linkedin.com/in/sayedjustetc

Blog: http://Bangla.SaLearningSchool.com, http://SitesTree.com
Online and Offline Training: http://Training.SitesTree.com (Also, can be free and low cost sometimes)

Facebook Group/Form to discuss (Q & A): https://www.facebook.com/banglasalearningschool

Our free or paid training events: https://www.facebook.com/justetcsocial

Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others. http://sitestree.com/training/

If you want to contribute to occasional free and/or low cost online/offline training or charitable/non-profit work in the education/health/social service sector, you can financially contribute to: safoundation at salearningschool.com using Paypal or Credit Card (on http://sitestree.com/training/enrol/index.php?id=114 ).

Optimization and Linear Algebra/Math from the Internet

First order taylor approximation formula?

https://www.thestudentroom.co.uk/showthread.php?t=1247928

Hessian Matrix

https://en.wikipedia.org/wiki/Hessian_matrix

"In mathematics, the Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field. It describes the local curvature of a function of many variables."

Use in optimization

"Hessian matrices are used in large-scale optimization problems within Newton-type methods because they are the coefficient of the quadratic term of a local Taylor expansion of a function. That is,

"

"Newton’s method in optimization"

In calculus, Newton’s method is an iterative method for finding the roots of a differentiable function F, which are solutions to the equation F (x) = 0. In optimization, Newton’s method is applied to the derivative f ′ of a twice-differentiable function f to find the roots of the derivative (solutions to f ′(x) = 0), also known as the stationary points of f. These solutions may be minima, maxima, or saddle points.[1]

https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization

SOLVING LINEAR DIFFERENTIAL EQUATIONS WITH THE LAPLACE TRANSFORM

https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118733639.app6

Pointwise supremum of a convex function collection

is it "I think it is either assumed that the ?? are defined on the same domain ?, or that (following a common convention) we set ??(?)=+∞ if ?∉Dom(??). You can easily check that under this convention, the extended ?? still remain convex and the claim is true."

https://math.stackexchange.com/questions/402919/pointwise-supremum-of-a-convex-function-collection?rq=1

"The supremum of a set is its least upper bound and the infimum is its greatest

upper bound."

https://www.math.ucdavis.edu/~hunter/m125b/ch2.pdf

Sine and Cosine Values

https://math.stackexchange.com/questions/1553990/easy-way-of-memorizing-values-of-sine-cosine-and-tangent/1554126

Barrier Function

"

Barrier function. In constrained optimization, a field of mathematics, a barrier function is a continuous function whose value on a point increases to infinity as the point approaches the boundary of the feasible region of an optimization problem.

"

https://en.wikipedia.org/wiki/Barrier_function

Trace: Marix

https://en.wikipedia.org/wiki/Trace_(linear_algebra)

Determinant

"The determinant of a matrix A is denoted det(A), det A, or |A|. Geometrically, it can be viewed as the volume scaling factor of the linear transformation described by the matrix. This is also the signed volume of the n-dimensional parallelepiped spanned by the column or row vectors of the matrix. The determinant is positive or negative according to whether the linear mapping preserves or reverses the orientation of n-space."

Ref: https://en.wikipedia.org/wiki/Determinant

Note: Older short-notes from this site are posted on Medium: https://medium.com/@SayedAhmedCanada

*** . *** *** . *** . *** . ***

Sayed Ahmed

BSc. Eng. in Comp. Sc. & Eng. (BUET)
MSc. in Comp. Sc. (U of Manitoba, Canada)
MSc. in Data Science and Analytics (Ryerson University, Canada)
Linkedin: https://ca.linkedin.com/in/sayedjustetc

Blog: http://Bangla.SaLearningSchool.com, http://SitesTree.com
Online and Offline Training: http://Training.SitesTree.com (Also, can be free and low cost sometimes)

Facebook Group/Form to discuss (Q & A): https://www.facebook.com/banglasalearningschool

Our free or paid training events: https://www.facebook.com/justetcsocial

Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others. http://sitestree.com/training/

If you want to contribute to occasional free and/or low cost online/offline training or charitable/non-profit work in the education/health/social service sector, you can financially contribute to: safoundation at salearningschool.com using Paypal or Credit Card (on http://sitestree.com/training/enrol/index.php?id=114 ).

SeDuMi MATLAB add-on: solve optimization problems with linear, quadratic and semidefiniteness constraints

"Abstract

SeDuMi is an add-on for MATLAB, which lets you solve optimization problems with linear, quadratic and semidefiniteness constraints. It is possible to have complex valued data and variables in SeDuMi. Moreover, large scale optimization problems are solved efficiently, by exploiting sparsity. This paper describes how to work with this toolbox."

https://www.tandfonline.com/doi/abs/10.1080/10556789908805766?journalCode=goms20

Note: Older short-notes from this site are posted on Medium: https://medium.com/@SayedAhmedCanada

*** . *** *** . *** . *** . ***

Sayed Ahmed

BSc. Eng. in Comp. Sc. & Eng. (BUET)
MSc. in Comp. Sc. (U of Manitoba, Canada)
MSc. in Data Science and Analytics (Ryerson University, Canada)
Linkedin: https://ca.linkedin.com/in/sayedjustetc

Blog: http://Bangla.SaLearningSchool.com, http://SitesTree.com
Online and Offline Training: http://Training.SitesTree.com (Also, can be free and low cost sometimes)

Facebook Group/Form to discuss (Q & A): https://www.facebook.com/banglasalearningschool

Our free or paid training events: https://www.facebook.com/justetcsocial

Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others. http://sitestree.com/training/

If you want to contribute to occasional free and/or low cost online/offline training or charitable/non-profit work in the education/health/social service sector, you can financially contribute to: safoundation at salearningschool.com using Paypal or Credit Card (on http://sitestree.com/training/enrol/index.php?id=114 ).

https://www.coursera.org/lecture/state-estimation-localization-self-driving-cars/lesson-1-the-linear-kalman-filter-7DFmY

https://d3c33hcgiwev3.cloudfront.net/gWbwrisXEem4egrIUlgmqg.processed/full/360p/index.webm?Expires=1579392000&Signature=gLd7RN8aqZhrNLNLl-huuNsIrkWnUp8gPUAMNqk6Xnkx0lmkMKE8XdXs5v7GGSMvq9ieVeR7MAi2bDz6pxUhgWspfMtnZZ2k2ZpKKzKdNoiFHW-zBVcnFTq~yPyC0ssd1gHzenk2SHqPBu1BhkHTqz7nhdXU08UQS-Z1w7qhwcw_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A

*** ***
Note: Older short-notes from this site are posted on Medium: https://medium.com/@SayedAhmedCanada

*** . *** *** . *** . *** . ***

Sayed Ahmed

BSc. Eng. in Comp. Sc. & Eng. (BUET)
MSc. in Comp. Sc. (U of Manitoba, Canada)
MSc. in Data Science and Analytics (Ryerson University, Canada)
Linkedin: https://ca.linkedin.com/in/sayedjustetc

Blog: http://Bangla.SaLearningSchool.com, http://SitesTree.com
Online and Offline Training: http://Training.SitesTree.com (Also, can be free and low cost sometimes)

Facebook Group/Form to discuss (Q & A): https://www.facebook.com/banglasalearningschool

Our free or paid training events: https://www.facebook.com/justetcsocial

Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others. http://sitestree.com/training/

If you want to contribute to occasional free and/or low cost online/offline training or charitable/non-profit work in the education/health/social service sector, you can financially contribute to: safoundation at salearningschool.com using Paypal or Credit Card (on http://sitestree.com/training/enrol/index.php?id=114 ).

“What is machine learning optimization?

Optimization is the most essential ingredient in the recipe of machine learning algorithms. It starts with defining some kind of loss function/cost function and ends with minimizing the it using one or the other optimization routine.Sep 5, 2018″
https://towardsdatascience.com/demystifying-optimizations-for-machine-learning-c6c6405d3eea

Ordered vector space

“Given a vector space V over the real numbers R and a preorder ≤ on the set V, the pair (V, ≤) is called a preordered vector space if for all x, y, z in V and 0 ≤ λ in R the following two axioms are satisfied

1. x ≤ y implies x + z ≤ y + z
2. y ≤ x implies λy ≤ λx.

If ≤ is a partial order, (V, ≤) is called an ordered vector space. The two axioms imply that translations and positive homotheties are automorphisms of the order structure and the mapping x ↦ −x is an isomorphism to the dual order structure. Ordered vector spaces are ordered groups under their addition operation.“
https://en.wikipedia.org/wiki/Ordered_vector_space

Algebra > Vector Algebra >

Vector Ordering

“If the first nonzero component of the vector difference is , then . If the first nonzero component of is , then .”

http://mathworld.wolfram.com/VectorOrdering.html

Vectors:

https://www.mathsisfun.com/algebra/vectors.html

Vector: Dot Product: Costheta

https://www.mathsisfun.com/algebra/vectors-dot-product.html

Vector Cross Product
https://www.mathsisfun.com/algebra/vectors-cross-product.html

“Optimization lies at the heart of machine learning. Most machine learning problems reduce to optimization problems.”
https://www.quora.com/What-is-the-relationship-between-machine-learning-and-mathematical-optimization

“Why is optimization important?
The purpose of optimization is to achieve the “best” design relative to a set of prioritized criteria or constraints. These include maximizing factors such as productivity, strength, reliability, longevity, efficiency, and utilization. … This decision-making process is known as optimization.”
https://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1031&context=ncete_publications

*** . *** . ***
Sayed Ahmed

BSc. Eng. in Comp. Sc. & Eng. (BUET)
MSc. in Comp. Sc. (U of Manitoba, Canada)
MSc. in Data Science and Analytics (Ryerson University, Canada)
Linkedin: https://ca.linkedin.com/in/sayedjustetc

Blog: http://Bangla.SaLearningSchool.com, http://SitesTree.com
Online and Offline Training: http://Training.SitesTree.com
FB Group on Learning/Teaching: https://www.facebook.com/banglasalearningschool
Our free or paid events on IT/Data Science/Cloud/Programming/Similar: https://www.facebook.com/justetcsocial

Get access to courses on Big Data, Data Science, AI, Cloud, Linux, System Admin, Web Development and Misc. related. Also, create your own course to sell to others. http://sitestree.com/training/

If you want to contribute to the operation of this site (Bangla.SaLearn) including occasional free and/or low cost online/offline training: http://Training.SitesTree.com (or charitable/non-profit work in the education/health/social service sector), you can financially contribute to: safoundation at salearningschool.com using Paypal or Credit Card (on http://sitestree.com/training/enrol/index.php?id=114 ).