Answer all 3 questions
This examination paper is worth a total of 60 marks.
(a) Given a collection of N documents D = {D1,…,DN}, your task is to implement a functionality that provides a list of suggested ‘more like this’ documents. With this problem context, answer the following questions.
(i) Explain how would you represent each document D ∈ D as a (real-valued) vector d.
What is the dimension of each vector? [2]
(ii) What does the L0 norm of a document vector indicate (in plain English) as per your definition of the document vectors in the previous question? [1]
(iii) How would you define the Lp distance between two document vectors d and d ′ ? [2]
(iv) What distance or similarity measure would you use for finding the set of ‘more like this’ documents for a current (given) document vector d, and why. [2]
(b) The probability distribution function of an n dimensional Gaussian is given by
f(x) = (x− µ) TΣ −1 (x− µ),
where µ ∈ R n is the mean vector and Σ ∈ R n×n is a square and invertible matrix, called the covariance matrix. Consider the particular case of n = 2. Answer the following questions.
(i) Plot the contours of the following Gaussians. For each contour plot, show the conditional distributions along the two axes.
µ = (0,0)
Σ = 1 00 1
µ = (0,0)
Σ = 0.5 00 2
µ = (0,0)
Σ = 0.5 0.10.5 2
µ = (0,0)
Σ = 0.5 0.1−0.5 2
[2](ii) Which one/ones of the above 4 Gaussian distributions can be reduced to a a single dimensional Gaussian with PCA on the covariance matrix without too much loss of information. Note that you do not need to explicitly compute the Eigenvalues. You should rather derive your answer from a visual interpretation of the contour plots.
Clearly explain your answer. [2]
(c) With respect to linear regression, answer the following questions.
(i) Derive the expression for stochastic gradient descent for linear regression with the squared loss function. Clearly introduce your notations for the input/output instances,and the parameter vector. [4]
(ii) Explain how linear regression can be extended to polynomial (higher order) regression?
What is the problem of using high degree polynomials for regression? How can that problem be alleviated? [3]
(iii) A common practice in stochastic gradient descent is to use a variable learning rate α for the parameter updates θj (t+1) ← θj (t) +α (t) ∂L ∂ θj , where θj (t) denotes the j th component of the parameter vector θ at iteration t, and α (t) denotes the value of the learning rate at iteration t. Which of the following alternatives of the learning rate update would you prefer (α is a constant) and why?
a)α (t) = α t , b)α (t) = α +t. [2]
Consider a card game where you have 4 suits (heart, diamonds, clubs and spades) and in each suit the cards 7, 8, 9, 10, Jack (J), Queen (Q), King (K) and Ace (A). In this question we will use the following commonly used terms:
(a) Assuming that you draw a single card at random from the pack, give the probabilities for the following events
(i) Drawing an Ace?
(ii) Drawing a red card?
(iii) Drawing a diamonds?
(iv) Drawing a royalty figure (Jack, Queen or King)?
(v) Drawing the Ace of spades?
[5]
(b) Now assume that you have already drawn the three cards: 10,J,Q. When drawing two more cards from the pack, what is the probability to obtain:
(i) A pair of two cards with the same value (eg, two Jacks).
(ii) Two pairs (eg, two Jacks and two Queens).
(iii) Three of a kind (eg, three Jacks).
(iv) A sequence of 5 cards (eg, 10, J, Q, K, A). Note that the cards can be of any suit, but there cannot be a break in the sequence.
[4]
(c) Now let us assume the following payout table for each hand of 5 cards: hand payout sequence of 5 cards 50
three of a kind 30
two pairs 20
one pair 10
anything else 0
As before, you have the cards 10, J, Q in hand.
(i) If you draw two more cards randomly from the deck, what is the expected value of the payout for this hand? [3]
