The hyper Dirichlet process and its discrete approximations:

the single conditional independence model ^{∗}

C. Asci ^{†}, G. Nappo^{‡}, M. Piccioni ^{§}

Abstract

The aim of this paper is the study of some laws of random probability distributions, called hyper Dirichlet processes, charging the product of three sample spaces, with the property that the first and the third components are independent conditional to the second one. The law of the marginals on the first two and on the last two components are specified to be Dirichlet processes with the same marginal parameter measure on the common second component. The joint law is then obtained as the hyper Markov combination of these two laws, as introduced in [3], and in fact these laws are generalizations of the hyper Dirichlet laws on contingency tables considered in the above paper.

Our main result is the convergence to the law of a hyper Dirichlet process of the sequence of hyper Dirichlet laws associated to finer and finer ”discretizations” of the two parameter measures, which is proved by means of a suitable coupling construction.

MSC: 60G57; 62G99; 62F15; 60F99. Keywords: Dirichlet laws and processes; Bayesian statistics; Markov distributions; weak convergence; coupling.

∗Partially supported by project ”Modelli probabilistici e applicazioni”

†Dipartimento di Studi geoeconomici, linguistici, statistici, storici per l’analisi regionale - Università di Roma ”La Sapienza”, via del Castro Laurenziano 9, I 00161 Roma, Italy

‡Dipartimento di Matematica - Università di Roma ”La Sapienza”, piazzale A. Moro 2, I 00185 Roma, Italy

§Dipartimento di Matematica - Università di Roma ”La Sapienza”, piazzale A. Moro 2, I 00185 Roma, Italy

1 Introduction

The hyper Markov combination of a family of laws for random probability measures has been defined by Dawid and Lauritzen [3]. Their construction starts from a general decomposable graph, whose vertices indicates the components of the product space supporting these random distributions. This graph represents the Markov structure imposed on the realizations, specifying the conditional independence relations assumed to hold between the components. The fact that the graph is decomposable implies that, under general assumptions guaranteeing the existence of regular conditional probabilities, a Markov distribution on the whole sample space can be uniquely derived from the marginal distributions of the components belonging to each clique: indeed, it is the Markov combination of these distributions, namely the only joint distribution with the specified marginals having the prescribed Markov structure. Clearly the specified marginals have to be consistent, in the sense that when two cliques intersect, the corresponding marginals have to agree on the common margins. It is then natural to build laws of random probability distributions with a prescribed Markov structure from the family of laws of their marginal distributions over each clique. Again, when two cliques intersect, the two corresponding random distributions have to induce the same law on the common margins, i.e. such a family have to be hyperconsistent. Clearly it remains the freedom to specify the joint law of all these marginals, but there is a unique one which has the hyper Markov property. This property is naturally obtained by ”lifting” the Markov property from the level of the realizations of random distributions to the level of their laws. In [3] this property is studied from various points of view, including its consequences for the Bayesian analysis of decomposable graphical models.

A particularly relevant example of this construction is the hyper Dirichlet law, which is the hyper Markov combination of hyperconsistent Dirichlet laws. Dirichlet laws are examples of random distributions over discrete sample spaces. For more general sample spaces their generalization is given by Dirichlet processes, introduced by Ferguson [5], and then studied by Blackwell and MacQueen [2] Sethuraman-Tiwari [11] and Sethuraman [10], between others. They still represent the most used class of prior distributions to deal with nonparametric models within the Bayesian framework. To our knowledge, the present paper is a first attempt to introduce genuinely multivariate properties, such as conditional independence relations, in the area of Bayesian nonparametric statistics.

In this paper we study hyper Dirichlet processes, which are hyper Markov combinations of hyperconsistent Dirichlet processes. In order to clarify the main ideas, we limit ourselves to consider a decomposable graph of the simplest kind, which consists in a non-trivial decomposition in two complete subgraphs. By merging vertices and edges in a natural way a graph of this type can be reduced to the graph with three vertices {1, 2, 3} with cliques {1, 2} and {2, 3}; hence 2 separates 1 and 3. This means that our aim is the construction of random probability distributions on the product of three sample spaces, with the property that the first and the third components are independent conditional to the second one. In the following the word Markov or hyper Markov is always referred to this graph. In order to construct a hyper Markov combination on this graph, we need to specify two hyperconsistent laws on the first and second components, and on the second and third components, respectively. When these are the laws of two Dirichlet processes, hyperconsistency is clearly equivalent to consistency of the two parameter measures. If the marginal parameter measure on the common second component is diffuse, a hyper Dirichlet process is itself a Dirichlet process (whose parameter measure is the Markov combination of the two given consistent parameter measures). This is surprising since hyper Dirichlet laws are never Dirichlet laws, except in trivial cases. This observation led us to study the relation between a hyper Dirichlet process and the hyper Dirichlet laws which are obtained by a natural discretization of its parameter measures. Our main result is that, under suitable assumptions, as the discretization refines, the hyper Dirichlet process appears to be the weak limit (i.e. in law) of the sequence of ”discretized” hyper Dirichlet laws.

Another contribution of the paper is the result that hyper Dirichlet processes are conjugate w.r.t. i.i.d. sampling, despite the fact that they are in general not strongly hyper Markov in the sense of [3]. This property is clearly useful for classes of nonparametric priors in Bayesian inference.

The next step of our analysis would be to consider more general decomposable graphs using the well-known fact that decomposable graphs can be always constructed through a series of decompositions [7]. It is not difficult to generalize the limit result when all the Dirichlet processes to be combined have consistent parameter measures with diffuse common marginals, in which case the hyper Markov combination is again a Dirichlet process, whose the parameter measure is the Markov combination of the given consistent parameter measures. However, in the general case some technical difficulties may arise, because such a Markov combination may have a complicated structure of the marginal atoms on the separators in the perfect decomposition of the graph.

The paper is organized in the following way. In the next section, after having reviewed the main properties of Dirichlet and hyper Dirichlet laws, we introduce the hyper Dirichlet process for pairs of parameter measures which have a common diffuse marginal. In Section 3 the convergence of hyper Dirichlet laws to this process is proved, over finer and finer discretizations. The basic tool is the Sethuraman representation of a Dirichlet process, together with a coupling construction between the limit process and the discretized one. In Section 4 we remove the assumption that the common marginal is diffuse, in which case we deal with hyper Dirichlet processes which are not in general Dirichlet processes. The main tool here is a suitable mixture decomposition of the Markov combination of the two parameter measures, which yields a corresponding mixture structure for the realizations of the process. We finally prove that hyper Dirichlet processes are conjugate with respect to random i.i.d. sampling, and discuss the relation of this conjugacy property with that holding for Dirichlet process ([5], [10]).

2 The hyper Dirichlet law and process for the simplest decomposition

Dirichlet laws are supported by the unit simplices in Euclidean spaces, hence they are used as laws for random variables with values in the set of probability distributions over a finite set of atoms L. For ease of notation the set L is always identified with {1, ..., |L|}, without loss of generality. Let us suppose first that |L| > 1. For a given vector {φ(l) > 0, l ∈ L}, we say that the random vector {G(l), l ∈ L} is Di {φ(l), l ∈ L} if

G(|L|) = 1 − ∑_{l=1}^{|L|−1} G(l)

and the joint density of (G(1), ..., G(|L| − 1)) w.r.t. the (|L| − 1)-dimensional Lebesgue measure on the set {g_{l} > 0, l = 1, ..., |L| − 1 and ∑_{i=1}^{|L|−1} g_{i} < 1} is

p(g_{1}, ..., g_{|L|−1}) = \frac{Γ(φ(L))}{∏_{l=1}^{|L|} Γ(φ(l))} ∏_{l=1}^{|L|−1} g_{l}^{φ(l)−1} (1 − ∑_{i=1}^{|L|−1} g_{i})^{φ(|L|)−1} ,

where φ(L) = ∑_{l=1}^{|L|} φ(l). In the sequel we will also allow φ(l) = 0 for some (but not all) index l, in which case we mean that G(l) ≡ 0. If there is only one index m ∈ L such that φ(m) > 0, then we define G(l) = δ_{l,m}, whatever the value φ(m) is. It is well known that if {G(l), l ∈ L} is Di {φ(l), l ∈ L} then, for any l ∈ L, G(l) is Beta(φ(l), φ(L)−φ(l)) (with the convention that a Beta(c, 0) is identically 1 for any c > 0 and a Beta(0, c) random variable is identically 0 for any c > 0), hence

E(G(l)) = \frac{φ(l)}{φ(L)} =: q(l), l ∈ L.  (1)

Next consider a non null contingency table

φ_{12} = {φ_{12}(i, j) ≥ 0, i ∈ I, j ∈ J}

and a random table

G_{12} ∼ Di (φ_{12})  (2)

over a finite product set I ×J. We use the standard notation for the marginal

φ_{2} (j) = ∑_{i∈I} φ_{12}(i, j)

and for the conditionals

φ_{1|2}(i|j) = \{ \begin{matrix}\frac{φ_{12}(i,j)}{φ_{2}(j)} & if φ_{2}(j) > 0, \\ 0 & if φ_{2}(j) = 0. \end{matrix}

These notations are used also for random probability distributions over I ×J.

By using the representation of the Dirichlet law as a function of independent Gamma variables (see e.g. Kotz, Balakrishnan, and Johnson [6]), the following fundamental result is immediately obtained.

Proposition 1 Let the table G_{12} ∼ Di(φ_{12}). Then the marginal vector G_{2} and the vectors of conditional distributions {G_{1|2}(·|j), j ∈ I : φ_{2}(j) > 0} are mutually independent, with

G_{2} ∼ Di (φ_{2}) ,  (3)
G_{1|2}(·|j) ∼ Di (φ_{12}(·, j)) , ∀j ∈ J : φ_{2}(j) > 0.  (4)

If φ_{2}(j) = 0 for some j ∈ J, then G_{12}(i, j) is identically zero for i ∈ I, in which case G_{1|2}(i|j) can be defined to be an arbitrary probability on I.

Conversely, if G_{12} is a random table with the properties (3) and (4), such that G_{12}(i, j) = 0 whenever φ_{2}(j) = 0, then G_{12} ∼ Di(φ_{12}).

Next consider another non null contingency table

φ_{23} = {φ_{23}(j, k), j ∈ J, k ∈ K}  (5)

where K is another finite set. We assume that φ_{12} and φ_{23} are consistent, i.e.

φ_{2}(j) = ∑_{i∈I} φ_{12}(i, j) = ∑_{k∈K} φ_{23}(j, k), for any j ∈ J.  (6)

Then the Markov combination φ of φ_{12} and φ_{23} is defined as

φ(i, j, k) = \{ \begin{matrix}\frac{φ_{12}(i,j)φ_{23}(j,k)}{φ_{2}(j)} & if φ_{2}(j) > 0, \\ 0 & otherwise. \end{matrix}

Notice φ is the only contingency table over I × J × K with the properties:

a) the marginals of φ over I ×J and J ×K equal to φ_{12} and φ_{23}, respectively;

b) φ(i, j, k) factorizes in a product of a function of (i, j) and one of (j, k).

Property b) implies that if φ is normalized it becomes a probability distribution over the product I × J × K under which the projections on the first and the third factor are independent conditional to the projection on the second one.

Now we construct a random probability distribution on the set I×J×K in the following way. Let G_{12} be distributed as in (2) and consider independent vectors

G_{3|2}(·|j) ∼ Di (φ_{23}(j, ·)) , j ∈ J : φ_{2}(j) > 0,  (7)

independent of G_{12}. Then define the three-dimensional random array

G = {G_{12}(i, j)G_{3|2}(k|j) for i ∈ I, j ∈ J, k ∈ K} .  (8)

Since G_{12}(i, j) = G_{2}(j)G_{1|2}(i|j), recalling (3) and (4), we observe that

i) the laws of the marginals over I × J and J × K are respectively

G_{12} ∼ Di (φ_{12}) , G_{23} = {G_{2}(j)G_{3|2}(k|j), j ∈ J, k ∈ K} ∼ Di (φ_{23}) ,

ii) the realizations {G(i, j, k), i ∈ I, j ∈ J, k ∈ K} factorize in a product of a function of (i, j) and one of (j, k), hence they are Markovian probability distributions; iii) the families of conditional distributions

{G_{1|2}(·|j), j ∈ J} , {G_{3|2}(·|j), j ∈ J}

and the marginal distribution G_{2} are mutually independent.

Property iii) is called the strong hyper Markov property [3].

The law of G is clearly uniquely determined by the properties i), ii) and iii). As such it is called the hyper Markov combination of the laws Di (φ_{12}) and Di (φ_{23}). Following [3] we call this law a hyper-Dirichlet law and indicate it by HDi(φ_{12}, φ_{23}).

The following proposition is worth of notice.

Proposition 2 The hyper Dirichlet law HDi(φ_{12}, φ_{23}) is a Dirichlet law if and only if, for any j such that φ_{2}(j) > 0, at least one of the following sets

I_{j} = {i : φ_{12}(i, j) > 0} , K_{j} = {k : φ_{23}(j, k) > 0} ,

is a singleton. In this case

HDi(φ_{12}, φ_{23}) = Di(φ).

Proof. If G is defined by (2), (7) and (8) then G_{2} ∼ Di (φ_{2}). Hence by Proposition 1 it is clear that G is Di(φ̃), for some contingency table φ̃, if and only if φ̃_{2}(j) = φ_{2}(j) for any j, and each of the conditionals

{G_{13|2}(i, k|j) = G_{1|2}(i|j)G_{3|2}(k|j), i ∈ I, k ∈ K} , j : φ_{2}(j) > 0, (9)

is Di({φ̃(i, j, k), i ∈ I, k ∈ K}). Then

φ̃(I × J × K) = φ̃_{2}(J) = φ_{2}(J) = φ(I × J × K).

By (1), for G ∼ Di(φ̃)

E [G(i, j, k)] = \frac{φ̃(i, j, k)}{φ̃(I × J × K)},

and, for G ∼ HDi(φ_{12}, φ_{23}) and φ_{2}(j) > 0

E [G(i, j, k)] = E [G_{12}(i, j)G_{3|2}(k|j)] = E [G_{12}(i, j)] E [G_{3|2}(k|j)]

= \frac{φ_{12}(i, j)}{φ_{12}(I × J)} \frac{φ_{23}(j, k)}{φ_{2}(j)} = \frac{φ(i, j, k)}{φ(I × J × K)},

hence it is immediately seen that necessarily φ̃ = φ.

Therefore, whenever φ_{2}(j) > 0, (9) is Di {φ(i, j, k), i ∈ I, k ∈ K}, from which its law is concentrated on a I_{j} × K_{j} − 1 manifold of R^{I_{j}×K_{j}} but not on any submanifold of lower dimension. On the other hand, by (4) and (7),

(9) is concentrated on a I_{j} + K_{j} − 2 dimensional manifold. Thus it has to be either I_{j} = 1 or K_{j} = 1, or both. The sufficiency of this condition is obvious.

It is clear that Dirichlet laws can be used only as laws for random distributions on a finite sample space L. For a general measurable sample space (L, L) we consider the space P(L) of all probability measures on (L, L), endowed with the smallest σ-algebra which makes the functions P ∈ P(L) → P (A) measurable, for all A ∈ L. A random distribution Γ on (L, L) is then a measurable mapping defined on some probability space with values in P(L). By Dynkin’s lemma the law of Γ is uniquely determined by the joint law of (Γ(A_{1}), ..., Γ(A_{m})) for all finite measurable partitions {A_{h}, h = 1, ..., m} of L. Given any finite measure φ on (L, L), we say that Γ is a Dirichlet process with parameter measure φ if this joint law is Di {φ(A_{h}), h = 1, ..., m}. The law of this process will be indicated by DI(φ), in order to distinguish it from the case of a finite space. The existence of a random distribution with this property is ensured by the following representation, introduced in [11] and further analyzed in [10].

In order to introduce this representation, first define T : (0, 1)^{∞} → (0, 1)^{∞} with

T_{l}(b) = b_{l} ∏_{i=1}^{l−1}(1 − b_{i}), l = 1, 2, ...

with the usual convention that a product over an empty set is equal to 1. If T is applied to a sequence β with i.i.d. components β_{l} ∼ Beta(1, φ(L)), l = 1, 2, ..., then with probability 1

γ = T(β) ∈ Δ_{∞} = {p ∈ (R^{+})^{∞} : ∑_{l} p_{l} = 1} ,

thus it defines a random distribution on ℕ^{+}. Next let us define the probability distribution Q = \frac{1}{φ(L)}φ on (L, L) and take

θ = {θ_{l}, l = 1, 2, ...} ,

with i.i.d. components θ_{l} ∼ Q. It can be shown that

κ(γ, θ) = ∑_{l} γ_{l}δ_{θ_{l}}  (10)

is a Dirichlet process with parameter measure φ, where κ is a measurable mapping of Δ_{∞} × L^{∞} into P(L). We thus write κ(γ, θ) ∼ DI(φ).

Next we extend the notion of Markov combination to distributions over more general spaces. In the sequel we assume that X, Y and Z are Borel spaces, i.e. they are measurably isomorphic to a Borel subset of a Polish space. This assumption ensures the existence of regular versions of conditional probabilities.

Next consider two consistent finite measures φ_{12} and φ_{23} on X × Y and Y × Z, respectively, i.e. such that

φ_{2}(dy) = φ_{12}(X × dy) = φ_{23}(dy × Z).

Thus the marginal measure φ_{2} is unambiguously defined. Our assumption implies that there exists two families of conditional measures {φ_{1|2}(·|y), y ∈ Y} and {φ_{3|2}(·|y), y ∈ Y} on X and Z respectively, such that

φ_{12}(dx, dy) = φ_{2}(dy)φ_{1|2}(dx|y), φ_{23}(dx, dy) = φ_{2}(dy)φ_{3|2}(dz|y).

Now we can construct the Markov combination φ of φ_{12} and φ_{23} on X × Y ×Z, namely

φ(dx, dy, dz) = φ_{1|2}(dx|y)φ_{2}(dy)φ_{3|2}(dz|y)  (11)
= φ_{12}(dx, dy)φ_{3|2}(dz|y) = φ_{1|2}(dx|y)φ_{23}(dy, dz).  (12)

From (12) we see that the marginals of φ on X × Y and Y × Z are φ_{12} and φ_{23}, respectively. Furthermore, for α = φ(X × Y × Z), the normalization Q = \frac{φ}{α} is Markovian, that is a draw θ = (U, V, W) from Q has the property that U and W are conditionally independent given V. These properties characterize φ uniquely.

Next let us consider a pair of laws 𝔓_{12} and 𝔓_{23} on P(X × Y) and P(Y × Z), respectively, which are hyperconsistent, in the sense they induce the same law of the marginal on Y. Then [3] there exists a unique law 𝔓 on P(X × Y × Z) such that, if the random distrubution P is drawn from 𝔓, and P_{12}, P_{23} and P_{2} are the corresponding marginals on X × Y, Y × Z and Y

1. P is Markovian w.p. 1;

2. P_{12} ∼ 𝔓_{12} and P_{23} ∼ 𝔓_{23};

3. P_{12} ⊥ P_{23}|P_{2}.

Properties i), ii) and iii) for X , Y, Z finite obviously imply 1, 2 and 3, respectively.

Property 3 is called the weak hyper Markov property and means specifically that the σ-algebra σ {P_{12}} generated by P (F × Z), for all measurable subsets F of X × Y and the σ-algebra σ {P_{23}} generated by P (X × G), for all measurable subsets G of Y × Z are independent conditionally to the σalgebra σ {P_{2}} generated by P (X × H × Z), for all measurable subsets H of Y. The law 𝔓 is then called the hyper Markov combination of 𝔓_{12} and 𝔓_{23}. Notice that property 3 replaces the stronger property iii), given in the discrete case.

Proposition 3 If φ_{2} has no atoms, then the law DI(φ) is the hyper Markov combination of DI(φ_{12}) and DI(φ_{23}).

Proof. We use the Sethuraman representation for DI(φ), i.e.

Γ = κ(γ, (U, V , W)) = ∑_{l} γ_{l}δ_{(U_{l},V_{l},W_{l})} ∼ DI(φ),  (13)

with (U_{l}, V_{l}, W_{l}), l = 1, 2, ... i.i.d. from Q, γ = T(β), β with i.i.d. components β_{l} ∼ Beta(1, α), l = 1, 2, ...

By computing the marginals

Γ_{12} = κ(γ, (U, V )), Γ_{23} = κ(γ, (V , W))

we check immediately that Γ_{12} ∼ DI(φ_{12}) and Γ_{23} ∼ DI(φ_{23}). Since φ_{2} has no atoms, the random sequence V = {V_{l}, l = 1, 2, ...} does not contain repetitions w.p. 1: this immediately implies that Γ is Markovian w.p. 1.

As for the hyper Markov property we observe that

σ {Γ_{2}} ⊂ σ {γ, V} , σ {Γ_{12}} ⊂ σ {γ, U, V} , σ {Γ_{23}} ⊂ σ {γ, V, W} ,

so that by the conditional independence of U and W given γ and V ,

Γ_{12} ⊥ Γ_{23} | γ, V.

Then, for any choice of measurable subsets {F_{i}, i = 1, ..., m} of X ×Y and {G_{j}, j = 1, ..., s} of Y × Z, for any pair h_{12} and h_{23} of bounded measurable functions defined on [0, 1]^{m} and [0, 1]^{s} respectively

E [h_{12}(Γ_{12}(F_{1}), ..., Γ_{12}(F_{m}))h_{23}(Γ_{23}(G_{1}), ..., Γ_{23}(G_{s})) | σ {γ, V}] = E [h_{12}(Γ_{12}(F_{1}), ..., Γ_{12}(F_{m}))| σ {γ, V}] E [h_{23}(Γ_{23}(G_{1}), ..., Γ_{23}(G_{s})) | σ {γ, V}] .

Thus it remains to prove that

E [h_{12}(Γ_{12}(F_{1}), ..., Γ_{12}(F_{m})) | σ {γ, V}] = E [h_{12}(Γ_{12}(F_{1}), ..., Γ_{12}(F_{m}))| σ {Γ_{2}}] , (15)
E [h_{23}(Γ_{23}(G_{1}), ..., Γ_{23}(G_{s})) | σ {γ, V}] = E [h_{23}(Γ_{23}(G_{1}), ..., Γ_{23}(G_{s}))| σ {Γ_{2}}] . (16)

If the function h_{12} is a polynomial, then (15) becomes

E [ m ; ∏_{l=1} (∑_{i} γ_{i}δ_{(U_{i},V_{i})}(F_{l}))^{d_{l}} | σ {γ, V }]

= ∑_{i_{1}}...∑_{i_{d}} d ; ∏_{k=1}γ_{i_{k}}δ_{V_{i_{k}}} (F_{l_{k},Y}) E [ d ; ∏_{k=1}δ_{U_{i_{k}}} (F_{l_{k},X}^{(V_{i_{k}})}) |σ {γ, V }]  (17)

where d = d_{1} + ⋯ + d_{m},

\{ \begin{matrix}l_{1}, …, l_{d_{1}} = 1 \\ l_{d_{1}+1}, …, l_{d_{1}+d_{2}} = 2 \\ … \\ l_{d_{1}+...+d_{m−1}+1}, …, l_{d} = m, \end{matrix}

and

F_{Y} = {y : ∃ (x, y) ∈ F } , F_{X}^{(y)} = {x : (x, y) ∈ F } ,

so that

δ_{x,y}(F) = δ_{y}(F_{Y})δ_{x}(F_{X}^{(y)}).

Then, taking into account that (U, V) are independent of γ and that (U_{i}, V_{i}) are i.i.d., we can rewrite the conditional expectation appearing in (17) as

E [δ_{U_{i_{1}}} (F_{l_{1},X}^{(V_{i_{1}})}) … δ_{U_{i_{d}}} (F_{l_{d},X}^{(V_{i_{d}})}) |σ {γ, V }]
= E [δ_{U_{i_{1}}} (F_{l_{1},X}^{(V_{i_{1}})}) … δ_{U_{i_{d}}} (F_{l_{d},X}^{(V_{i_{d}})}) |σ {V }]
= ψ (V_{i_{1}}, ⋯, V_{i_{d}}) ,

where ψ is the bounded measurable function

ψ (v) := E [δ_{U_{1}} ( ⋂_{i∈A_{1}(v)} F_{l_{i},X}^{(v′_{1})}) |V_{1} = v′_{1}] … E [δ_{U_{1}} ( ⋂_{i∈A_{f} (v)} F_{l_{i},X}^{(v′_{f})}) |V_{1} = v′_{f} ] ,

with the definition

A_{j}(v) = {i : v_{i} = v′_{j}},  j = 1, ⋯, f  (18)

where {v′_{1}, ⋯, v′_{f}} = {v_{1}, ⋯, v_{d}} is the set whose elements are the distinct components of the vector v = (v_{1}, ⋯, v_{d}).

Summarizing (17) can be written as

∑_{i_{1}}...∑_{i_{d}}γ_{i_{1}...}γ_{i_{d}}δ_{V_{i_{1}}} (F_{l_{1},Y}) ...δ_{V_{i_{d}}} (F_{l_{d},Y}) ψ (V_{i_{1}}, ⋯, V_{i_{d}})

= ∫_{F_{l_{1},Y}} ⋯ ∫_{F_{l_{d},Y}} ψ (y_{1}, ⋯, y_{d}) Γ_{2}(dy_{1}) ⋯ Γ_{2}(dy_{d}).

Arguing as in [4], it is readily proved that the above is a σ {Γ_{2}} −measurable function because the Borel σ-algebra of a Borel space is countably generated. This ends the proof of (15). Likewise we prove (16) when h_{23} is a polynomial. Finally observe that expectations of polynomials determine uniquely the joint conditional law of {Γ_{12}(F_{i}), i = 1, ..., m} and {Γ_{23}(G_{j}), j = 1, ..., s}, hence the whole proof is finished.

Following Dawid and Lauritzen we say that a random distribution is a hyper Dirichlet process with parameter measures φ_{12} and φ_{23} if it has the same law as Γ defined in (13). We denote this law by HDI(φ_{12}, φ_{23}) in order to distinguish it from the finite case, where we used the symbol HDi instead. The striking feature of the hyper Dirichlet process, when φ_{2} is diffuse, is that it is a Dirichlet process, namely HDI(φ_{12}, φ_{23}) = DI(φ), where φ is the Markov combination of φ_{12} and φ_{23}. On the contrary we saw that the hyper Dirichlet law is practically never a Dirichlet law. The aim of the next section is to prove that nonetheless, under suitable regularity conditions, we can exhibit a sequence of hyper Dirichlet laws converging in a natural sense to the law of a hyper Dirichlet process. In Section 4 we will turn our attention to more general hyper Dirichlet processes which are not Dirichlet process anymore.

3 Coupling the limiting process with the discrete approximations

In this section we take X = ℝ^{d_{1}}, Y = ℝ^{d_{2}} and Z = ℝ^{d_{3}}, and we assume that for each of these spaces we have a sequence of finite measurable partitions

{B_{i}^{(n)}, i ∈ I^{(n)}} , {C_{j}^{(n)}, j ∈ J^{(n)}} , {D_{k}^{(n)}, k ∈ K^{(n)}} , n = 1, 2, ..., (19)

respectively. Correspondingly, we discretize the consistent parameter measures φ_{12} and φ_{23} along the corresponding sequences of product partitions in two-dimensional cells

{B_{i}^{(n)} × C_{j}^{(n)}, i ∈ I^{(n)}, j ∈ J^{(n)}} , {C_{j}^{(n)} × D_{k}^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}} , (20)

getting the tables

φ_{12}^{(n)} = {φ_{12}(B_{i}^{(n)} × C_{j}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}}

(21)

φ_{23}^{(n)} = {φ_{23}(C_{j}^{(n)} × D_{k}^{(n)}), j ∈ J^{(n)}, k ∈ K^{(n)}} .

For each integer n consider a random array

G^{(n)} ∼ HDi(φ_{12}^{(n)}, φ_{23}^{(n)})  (22)

of probabilities G^{(n)}(i, j, k) randomly assigned to the three-dimensional cells B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)} for i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}. It is then natural to compare its law with that of the corresponding array of probabilities

Γ^{(n)}(i, j, k) = {Γ(B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}} . (23)

assigned by the process Γ ∼ HDI(φ_{12}, φ_{23}), which is a random distribution on ℝ^{d}, with d = d_{1} + d_{2} + d_{3}. This will be done by constructing both three-dimensional random arrays on the same probability space where the representation (13) of the hyper Dirichlet process Γ is defined, coupling them in a suitable way. First of all notice that whenever φ_{2}^{(n)}(j) = 0 it has to be Γ^{(n)}(i, j, k) = G^{(n)}(i, j, k) = 0 for any i ∈ I^{(n)} and k ∈ K^{(n)}, so we can assume w.l.o.g. that φ_{2}^{(n)}(j) > 0 for all j ∈ J^{(n)}. Next observe that

Γ^{(n)}(i, j, k) = ∑_{l}^{∗}γ_{l}, i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)},  (24)

where in ∑_{l}^{∗} the index l ranges in {l : (U_{l}, V_{l}, W_{l}) ∈ B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}}. Since

Γ_{12}^{(n)} ∼ Di (φ_{12}^{(n)})

we can define

G^{(n)}(i, j, k) = Γ_{12}^{(n)}(i, j)G_{3|2}^{(n)}(k|j), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)},  (25)

provided the vector {G_{3|2}^{(n)}(k|j), k ∈ K^{(n)}} is defined, independently for all

j ∈ J^{(n)} and independently of Γ_{12}, with the law Di {φ_{23}^{(n)}(j, k), k ∈ K^{(n)}}.

This can be achieved by using mutually independent (w.r.t. j ∈ J^{(n)}) Dirichlet processes on Z with parameter measure φ_{23}(C_{j}^{(n)} × ·) defined through the Sethuraman representation

∑_{r} H_{j,r}^{(n)}δ_{W_{j,r}^{(n)}},  (26)

where H_{j}^{(n)} = {H_{j,r}^{(n)}, r = 1, 2, ...} = T (β_{j}^{(n)}), with β_{j}^{(n)} = {β_{j,r}^{(n)}, r = 1, 2, ...}

i.i.d. Beta(1, φ_{2}^{(n)}(j)), and {W_{j,r}^{(n)}, r = 1, 2, ...} is a sequence of i.i.d. random

variables, with common distribution φ_{3|2}^{(n)}(·|j) given by

φ_{3|2}^{(n)}(dz|j) = \frac{φ_{23}(C_{j}^{(n)} × dz)}{φ_{2}(C_{j}^{(n)})}.

With these positions we define

G_{3|2}^{(n)}(k|j) = ∑_{r} H_{j,r}^{(n)}1_{D_{k}}(W_{j,r}^{(n)}), k ∈ K^{(n)},  (27)

All the random variables used to produce the vectors {G_{3|2}^{(n)}(k|j), k ∈ K^{(n)}}, for j ∈ J^{(n)}, are drawn independently of Γ, except for the mutually independent random variables W_{j,1}^{(n)}, j ∈ J^{(n)}. Each of these random variables is coupled with W_{l(j)}, where l(j) is the first index l (existing w.p. 1) such that V_{l} ∈ C_{j}. We choose the maximal coupling (also known as γ-coupling, see e.g. Lindvall [8]), conditional to V_{l(j)}, so that

P (W_{j,1}^{(n)} ≠ W_{l(j)}|V_{l(j)}) = \frac{1}{2} ∥φ_{3|2}(·|V_{l(j)}) − φ_{3|2}^{(n)}(·|j)∥ .  (28)

Having completed the construction, we are now ready to prove the following result.

Theorem 4 Under the following assumptions:

i) the marginal measure φ_{2} has no atoms;

ii) for any compact set A ⊂ Y

lim_{n→∞} max {|C_{j}^{(n)}| , j ∈ J^{(n)} : C_{j}^{(n)} ∩ A ≠ ∅} = 0,

where | · | is the diameter;

iii) the function y ∈ Y → φ_{3|2}(·|y) has a version which is continuous in total variation;

then

lim_{n→∞} ∑_{i∈I^{(n)}} ∑_{j∈J^{(n)}} ∑_{k∈K^{(n)}} E|Γ^{(n)}(i, j, k) − G^{(n)}(i, j, k)| = 0.

Proof. Along the whole proof we will drop the superscript n, except when we exhibit the final estimates. By standard manipulations we get

|Γ(i, j, k) − G(i, j, k)| = |Γ(i, j, k) − Γ_{12}(i, j)G_{3|2}(k|j)|  (29)

≤ γ_{l(j)}1_{B_{i}}(U_{l(j)}) |1_{D_{k}}(W_{l(j)}) − 1_{D_{k}} (W_{j,1})|

+ γ_{l(j)}1_{B_{i}}(U_{l(j)}) ∑_{r≥2} H_{j,r} |1_{D_{k}}(W_{j,1}) − 1_{D_{k}}(W_{j,r})|

+ ∑_{l>l(j)} γ_{l}1_{B_{i}}(U_{l})1_{C_{j}}(V_{l}) |1_{D_{k}}(W_{l}) − ∑_{r≥1} H_{j,r}1_{D_{k}}(W_{j,r})| ,

from which, by summing over i, j, k, recalling that ∑_{r≥2} H_{j,r} = 1 − H_{j,1} and that for any pair of probability distributions μ and ν over K

∑_{k} |μ(k) − ν(k)| = ∥μ − ν∥ ≤ 2,

we get

∑_{i,j,k} |Γ(i, j, k) − G(i, j, k)| ≤ ∑_{j} γ_{l(j)} ∑_{k} |1_{D_{k}}(W_{l(j)}) − 1_{D_{k}}(W_{j,1})|

+ 2 ∑_{j} γ_{l(j)}(1 − H_{j,1}) + 2 ∑_{j} ∑_{l>l(j)} γ_{l}1_{C_{j}}(V_{l}).  (30)

Let us proceed to bound the first term at the r.h.s. of (30). Since

∑_{k} |1_{D_{k}}(W_{l(j)}) − 1_{D_{k}}(W_{j,1})| ≤ 2 · 1_{{W_{l(j)}≠W_{j,1}}},

for any compact set A ⊂ Y we have

∑_{j} γ_{l(j)} ∑_{k} |1_{D_{k}}(W_{l(j)}) − 1_{D_{k}}(W_{j,1})| ≤ 2 ∑_{j:C_{j}∩A≠∅} γ_{l(j)}1_{{W_{l(j)}≠W_{j,1}}} + 2Γ_{2}(A^{c}),

and by taking the expected value and using (28), by the independence of γ, l(j) and V_{l(j)} we have

E ∑_{j} γ_{l(j)} ∑_{k} |1_{D_{k}}(W_{l(j)}) − 1_{D_{k}}(W_{j,1})|

≤ ∑_{j:C_{j}∩A≠∅} E [γ_{l(j)}] E [∥φ_{3|2}^{(n)}(·|j) − φ_{3|2}(·|V_{l(j)})∥] + 2E(Γ_{2}(A^{c}))

≤ sup_{j:C_{j}∩A≠∅} sup_{v∈C_{j}} ∥φ_{3|2}^{(n)}(·|j) − φ_{3|2}(·|v)∥ + \frac{2}{α}φ_{2}(A^{c}).  (31)

Now

sup_{v∈C_{j}} ∥φ_{3|2}^{(n)}(·|j) − φ_{3|2}(·|v)∥ = sup_{v∈C_{j}} ∥\frac{∫_{C_{j}}φ_{3|2}(·|y)φ_{2}(dy)}{φ_{2} (C_{j})} − φ_{3|2}(·|v)∥

= 2 sup_{v∈C_{j}} sup_{D} \frac{|∫_{C_{j}} (φ_{3|2}(D|y) − φ_{3|2}(D|v)) φ_{2}(dy)|}{φ_{2} (C_{j})}

≤ sup_{v∈C_{j}} \frac{∫_{C_{j}} ∥φ_{3|2}(·|y) − φ_{3|2}(·|v)∥ φ_{2}(dy)}{φ_{2} (C_{j})}

≤ sup_{v_{1},v_{2}∈C_{j}} ∥φ_{3|2}(·|v_{1}) − φ_{3|2}(·|v_{2})∥ .

Finally select the compact set A in such a way that φ_{2}(A^{c}) ≤ \frac{αε}{4} and define the larger compact set

A^{1} ≡ {t : dist(t, A) ≤ 1} .

From the assumption iii), on the compact set A^{1} the function y → φ_{3|2}(·|y) is uniformly continuous in total variation, so let δ(ε) < 1 such that

sup_{\begin{matrix}v_{1},v_{2}∈A^{1} \\ |v_{1}−v_{2}|<δ(ε) \end{matrix}} ∥φ_{3|2}(·|v_{1}) − φ_{3|2}(·|v_{2})∥ < \frac{ε}{2}.

From the assumption ii), for n large enough and any j such that C_{j}^{(n)}∩A ≠

∅, it holds |C_{j}^{(n)}| < δ(ε) so that C_{j}^{(n)} ⊂ A^{1}, and consequently the r.h.s. of (31) can be bounded by ε.

For the remaining terms we need first to observe that

λ_{n} := max_{j∈J^{(n)}} φ_{2}(C_{j}^{(n)})

tends to zero as n → ∞. If this is not true, by tightness of φ_{2} then we would have a sequence {C_{j_{n}}^{(n)}} of Borel sets of diameter shrinking to zero such that φ_{2}(C_{j_{n}}^{(n)}) > ε > 0 all contained in a given compact set. But then this sequence would accumulate at some point, thus denying the absence of atoms.

The expected value of the second term at the r.h.s. of (30) can now be bounded by

2 ∑_{j} E [γ_{l(j)}] (1 − E [H_{j,1}]) = 2 ∑_{j} E [γ_{l(j)}] \frac{φ_{2}(C_{j})}{1 + φ_{2}(C_{j})}

≤ 2 max_{j} \frac{φ_{2}(C_{j}^{(n)})}{1 + φ_{2}(C_{j}^{(n)})} ≤ 2λ_{n} ,

which tends to zero as n → ∞.

Observe that l(j) has a geometric distribution with probability of success p_{j} = \frac{φ_{2}(C_{j})}{α}, whereas

E(γ_{l(j)}|l(j) = h) = E(γ_{h}) = \frac{1}{1 + φ_{2}(C_{j})} (\frac{φ_{2}(C_{j})}{1 + φ_{2}(C_{j})})^{h−1} , h = 1, 2, ...

so that, by a trivial computation

E(γ_{l(j)}) = \frac{φ_{2}(C_{j})}{α + φ_{2}(C_{j})^{2}} ≥ \frac{φ_{2}(C_{j})}{α (1 + φ_{2}(C_{j}))}.

Finally observe that the expected value of the last sum at the r.h.s. of (30) is

2E [∑_{j} ∑_{l>l(j)} γ_{l}1_{C_{j}}(V_{l})] = 2E(1 − ∑_{j} γ_{l(j)})

≤ \frac{2}{α} ∑_{j} (φ_{2}(C_{j}) − \frac{φ_{2}(C_{j})}{1 + φ_{2}(C_{j})}) = \frac{2}{α} ∑_{j} \frac{φ_{2}(C_{j})^{2}}{1 + φ_{2}(C_{j})}

≤ \frac{2}{α} max_{j} φ_{2}(C_{j}^{(n)}) ∑_{j} \frac{φ_{2}(C_{j}^{(n)})}{1 + φ_{2}(C_{j}^{(n)})} ≤ 2λ_{n},

which again tends to zero as n → ∞.

The reader will expect that it is possible to formulate the result of the previous theorem as the convergence of G^{(n)} to Γ in a suitable sense. In order to use weak convergence we have to extend G^{(n)} to be a random distribution on the whole space ℝ^{d}. However in order to obtain the next result such an extension is essentially arbitrary.

Theorem 5 Under the assumption of the previous theorem, let G^{(n)} be any random probability distribution on X × Y ×Z = ℝ^{d}, with the property that

{G^{(n)}(B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}}

has the same law as

{G^{(n)}(i, j, k), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}} .

Furthermore suppose that for all compact sets E ⊂ X and F ⊂ Z

lim_{n→∞} max {|B_{i}^{(n)}| , i ∈ I^{(n)} : B_{i}^{(n)} ∩ E ≠ ∅} = 0  (32)
lim_{n→∞} max {|D_{k}^{(n)}| , k ∈ K^{(n)} : D_{k}^{(n)} ∩ F ≠ ∅} = 0.  (33)

Then {G^{(n)}} converges weakly to Γ, as random variables with values in the space of probability distributions on ℝ^{d}, endowed with the topology of weak convergence.

Proof. In order to prove that {G^{(n)}} converges weakly to Γ, it suffices to show that

lim_{n→∞}E [Φ (G^{(n)})] = E [Φ (Γ)] ,

for all functions Φ which are bounded and Lipschitz (BL) w.r.t. a metric dist that induces the topology of weak convergence on P(R^{d}), i.e. with the property that for all probability distributions μ and ν

|Φ (μ) − Φ (ν)| ≤ L_{Φ}dist (μ, ν) ,

with L_{Φ} < ∞ (see [1]).

In particular (see [9]) we can choose dist = dist_{BL}, i.e.

dist_{BL} (μ, ν) = sup_{∥f∥_{∞}≤1,L(f)≤1} |μ(f) − ν(f)| ,

where the supremum is taken over all bounded Lipschitz functions on R^{d} such that both the L^{∞} norm ∥f∥_{∞} and the Lipschitz constant L(f) are bounded by 1.

By assumption

G^{(n)}(dx, dy, dz) = ∑_{i,j,k} G^{(n)}(i, j, k)λ_{ijk}^{(n)}(dx, dy, dz),

where λ_{ijk}^{(n)} is a probability measure supported by B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}, for i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}. Likewise we define the intermediate random probability distribution on R^{d}, namely

Γ^{(n)}(dx, dy, dz) = ∑_{i,j,k} Γ^{(n)}(i, j, k)λ_{ijk}^{(n)}(dx, dy, dz),

where we recall that

Γ^{(n)}(i, j, k) = Γ(B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}).  (34)

Obviously

|E [Φ (G^{(n)})] − E [Φ (Γ)]|
≤ |E [Φ (G^{(n)}) − Φ (Γ^{(n)})]| + |E [Φ (Γ^{(n)}) − Φ (Γ)]| .

If Φ is BL, then

|E [Φ (G^{(n)})] − E [Φ (Γ^{(n)})]| ≤ E [L_{Φ}dist_{BL} (G^{(n)}, Γ^{(n)})]

≤ \frac{1}{2} L_{Φ}E [∥G^{(n)} − Γ^{(n)}∥] .

By Theorem 4 the variation distance between G^{(n)} and Γ^{(n)} tends to zero, hence it follows

lim_{n→∞} |E [Φ (G^{(n)})] − E [Φ (Γ^{(n)})]| = 0.

Furthermore

|E [Φ (Γ^{(n)})] − E [Φ (Γ)]| ≤ L_{Φ}E [dist_{BL} (Γ^{(n)}, Γ)]

= L_{Φ}E [ sup_{∥f∥_{∞}≤1,L(f)≤1} |Γ^{(n)}(f) − Γ(f)|] .

Since the quantity inside the expectation is bounded by 2, by the bounded convergence theorem, the statement will be proved once we show that for any probability measure Γ on ℝ^{d}

lim_{n→+∞} sup_{∥f∥_{∞}≤1,L(f)≤1} |Γ^{(n)}(f) − Γ(f)| = 0.

For this observe that

|Γ^{(n)}(f) − Γ(f)|

= |∑_{i,j,k} Γ^{(n)}(i, j, k) ∫_{B_{i}^{(n)}} ∫_{C_{j}^{(n)}} ∫_{D_{k}^{(n)}}f(x, y, z) (λ^{(n)}(dx, dy, dz) − \frac{Γ(dx, dy, dz)}{Γ^{(n)}(i, j, k)})| .

Let ϵ > 0 and consider compact sets A_{h} ⊂ R^{d_{h}} for h = 1, 2, 3 such that Γ (A^{c}) < ϵ, where A = A_{1} × A_{2} × A_{3}. Then, by splitting the sum as the sum ∑_{i,j,k}^{∗} over those indices (i, j, k) such that B_{i}^{(n)} ∩ A_{1} ≠ ∅, C_{j}^{(n)} ∩ A_{2} ≠ ∅ and D_{k}^{(n)} ∩ A_{3} ≠ ∅, and the sum over the remaining indices, the above display is bounded from above by

∑_{i,j,k}^{∗}Γ^{(n)}(i, j, k) | ∫_{B_{i}^{(n)}} ∫_{C_{j}^{(n)}} ∫_{D_{k}^{(n)}}f(x, y, z) (λ^{(n)}(dx, dy, dz) − \frac{Γ(dx, dy, dz)}{Γ^{(n)}(i, j, k)})|

+ 2 ∥f∥_{∞} ϵ

≤ ∑_{i,j,k}^{∗}Γ^{(n)}(i, j, k) ( ∫_{B_{i}^{(n)}} ∫_{C_{j}^{(n)}} ∫_{D_{k}^{(n)}} |f(x, y, z) − f (\overline{x}_{i}, \overline{y}_{j}, \overline{z}_{k})| λ^{(n)}(dx, dy, dz) (35)

+ ∫_{B_{i}^{(n)}} ∫_{C_{j}^{(n)}} ∫_{D_{k}^{(n)}} |f(x, y, z) − f (\overline{x}_{i}, \overline{y}_{j}, \overline{z}_{k})| \frac{Γ(dx, dy, dz)}{Γ^{(n)}(i, j, k)}) + 2ϵ,

for any choice of (\overline{x}_{i}, \overline{y}_{j}, \overline{z}_{k}) ∈ B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}.

Observe that from the assumption ii) of the previous Theorem 4, (32) and (33), for n large enough,

|B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}| < ϵ,

for all the indices (i, j, k) considered in the above sum. Hence for any (x, y, z) ∈ B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)} as above

|f(x, y, z) − f (\overline{x}_{i}, \overline{y}_{j}, \overline{z}_{k})| ≤ L(f)dist ((x, y, z), (\overline{x}_{i}, \overline{y}_{j}, \overline{z}_{k})) ≤ ϵ,

which implies that (35) does not exceed 4ϵ. Since ϵ is arbitrary, the proof is finished.

Remark 6 We remark that assumption iii) of Theorem 4 holds true when φ_{23} has a jointly continuous density g_{23}(y, z) w.r.t. to a product measure μ_{2} × μ_{3}, with the property that for any y ∈ ℝ^{d_{2}} there exists a μ_{3}-integrable function h_{y}(z), such that for any y′ in a neighborhood of y

g_{23}(y′, z) ≤ h_{y}(z) for all z ∈ ℝ^{d_{3}}.

In fact this implies that as y_{n} tends to y,

lim_{n→∞} ∫ |g_{23}(y_{n}, z) − g_{23}(y, z)|μ_{3}(dz) = 0,

the density g_{2}(y) = ∫ g_{23}(y, z)μ_{3}(dz) of φ_{2} w.r.t. μ_{2} is continuous in y, and by Bayes formula

∥φ_{3|2}(·|y_{n}) − φ_{3|2}(·|y)∥ ≤ \frac{2}{g_{2}(y)} ∫ |g_{23}(y_{n}, z) − g_{23}(y, z)|μ_{3}(dz),

for y in the open set of full φ_{2}-measure where g_{2}(y) > 0.

In the following sections we will prove results analogous to Theorem 4 and Theorem 5 for more general hyper Dirichlet processes.

4 More general hyper Dirichlet processes

In this section we construct a general hyper Dirichlet process by removing the assumption that the common marginal of its two parameter measures is diffuse. We start with the following general result about Dirichlet processes.

Proposition 7 Let us consider a sequence of finite measures {ρ^{(l)}, l = 0, 1, 2, ..} on the measurable space (E, E), and suppose that ρ(E) = ∑_{l} ρ^{(l)}(E) < +∞, in which case ρ is a finite measure as well. Let R = {R_{l}, l = 0, 1, 2, ...} be a sequence of independent random variables, with

R_{l} ∼ Be(ρ^{(l)}(E), ∑_{i=l+1}^{∞} ρ^{(i)}(E)),  l = 0, 1, 2, ...  (36)

and define the random sequence S = T(R), i.e.

S_{l} = R_{l} ∏_{i=0}^{l−1}(1 − R_{i}), l = 0, 1, 2, ...

Then i) S is a Dirichlet process on ℕ, with parameter measure ∑_{l} ρ^{(l)}(E)δ_{l}.

Moreover let {Γ_{l} ∼ DI (ρ^{(l)}) , l = 0, 1, 2, ...} be a sequence of independent Dirichlet processes on E, all independent of S. Then

ii) the random probability distribution on (E, E)

∑_{l=0}^{∞} S_{l}Γ_{l}  (37)

is a Dirichlet process on E with parameter measure ρ.

Proof. A Dirichlet process whose parameter measure is a single mass on some (fixed) point x is with probability 1 a Dirac probability distribution on x. As a consequence statement i) is included in statement ii), when ρ^{(l)} = ρ^{(l)}(E)δ_{l}, and therefore we prove only the latter.

Next observe that the residual masses 1−∑_{l=0}^{m} S_{l} of the sequence S decrease

to zero a.s. as m → ∞, since

E [1 − ∑_{l=0}^{m} S_{l}] = ∏_{l=0}^{m} E(1 − R_{l}) = ∏_{l=0}^{m} \frac{∑_{j=l+1}^{∞} ρ^{(j)}(E)}{∑_{j=l}^{∞} ρ^{(j)}(E)} = \frac{∑_{j=m+1}^{∞} ρ^{(j)}(E)}{ρ(E)} → 0

as m → ∞. Thus (37) is well defined as a random variable with values in the set of probability measures over (E, E).

Using the converse part of Proposition 1, and induction on m, it is not difficult check that

(S_{0}, ..., S_{m}, 1 − ∑_{l=0}^{m} S_{l}) ∼ Di (ρ^{(0)}(E), ..., ρ^{(m)}(E), ∑_{l=m+1}^{∞} ρ^{(l)}(E)) , (38)

for any integer m.

Let x_{0} be any fixed element of E and define the random probability distribution

{∑_{l=0}^{m} S_{l}Γ_{l} + (1 − ∑_{l=0}^{m} S_{l}) δ_{x_{0}}, h ∈ H}  (39)

Next consider any finite measurable partition {A_{h}, h ∈ H} of E, and suppose for the moment that the joint law of

{∑_{l=0}^{m} S_{l}Γ_{l}(A_{h}) + (1 − ∑_{l=0}^{m} S_{l}) δ_{x_{0}}(A_{h}), h ∈ H}  (40)

is Di (∑_{l=0}^{m} ρ^{(l)}(A_{h}) + ( ∑_{l=m+1}^{∞} ρ^{(l)}(E)) δ_{x_{0}}(A_{h}), h ∈ H).

If we let m → ∞ the random vector (40) converges a.s. to { ∑_{l=0}^{∞} S_{l}Γ_{l}(A_{h}), h ∈ H}, hence it converges in law as well. By the continuity of the Dirichlet laws w.r.t. their parameter vectors, the limit law is Di {∑_{l=0}^{∞} ρ^{(l)}(A_{h})}. This ends the proof of statement ii).

In order to see that (40) has the correct law, define the following twodimensional table

D_{12}(h, l) = S_{l}Γ_{l}(A_{h}),  h ∈ H, l = 0, ..., m,
D_{12}(h, m + 1) = (1 − ∑_{l=0}^{m} S_{l}) δ_{x_{0}}(A_{h}),  h ∈ H,

and observe that the marginal D_{2}(·) is (38) and the conditionals D_{1|2}(·|l) are {Γ_{l}(A_{h}), h ∈ H} for l = 0, ..., m, and {δ_{x_{0}}(A_{h}), h ∈ H} for l = m + 1. Therefore by Proposition 1

D_{12} ∼ Di(d_{12}(h, l), h ∈ H, l = 0, 1, ..., m + 1),

where

d_{12}(h, l) = ρ^{(l)}(A_{h}),  h ∈ H, l = 0, ..., m,
d_{12}(h, m + 1) = ( ∑_{l=m+1}^{∞} ρ^{(l)}(E)) δ_{x_{0}}(A_{h}),  h ∈ H.

Finally observe that the marginal D_{1} coincides with (40) and apply again Proposition 1 to conclude.

The consequence of the above proposition is the following convenient representation of a general two-dimensional Dirichlet process.

Corollary 8 Let ρ be a finite measure on the product space X × Y, and let {y_{l}, l = 1, 2, ...} be the set of the atoms of ρ_{2}(dy). Define the measure ψ as

ψ(dx, dy) := ∑_{l} δ_{{y_{l}}}(dy)ρ(dx × {y_{l}})

and the measure φ in such a way that ρ = φ + ψ.

Then the Dirichlet process with parameter measure ρ can be represented as

S_{0}Γ(dx, dy) + ∑_{l} S_{l}Ξ_{l}(dx)δ_{y_{l}}(dy),

where the sequence S = T(R), R being a sequence of independent random variables with

R_{0} ∼ Be(φ_{2}(Y), ψ_{2}(Y)), R_{l} ∼ Be(ψ_{2}({y_{l}}), ∑_{i=l+1}^{∞} ψ_{2}({y_{i}})), l = 1, 2, ..., (41)

Γ is a Dirichlet process on X × Y with parameter measure φ, and Ξ_{l} is a Dirichlet process on X with parameter measure ψ^{(l)}(dx) := ψ(dx × {y_{l}}), for l = 1, 2...., all mutually independent.

Proof. It suffices to apply the representation of Proposition 7 to the Dirichlet process on E = X × Y with parameter measure ρ = ∑_{l=0}^{∞} ρ^{(l)}, where

ρ^{(0)}(dx, dy) = φ(dx, dy), ρ^{(l)}(dx, dy) = ψ^{(l)}(dx)δ_{y_{l}}(dy), l = 1, 2, ...

with Γ_{0} = Γ and Γ_{l} ∼ DI (ψ^{(l)}(dx)δ_{y_{l}}(dy)), which can be represented as Ξ_{l}(dx)δ_{y_{l}}(dy), for l = 1, 2, ...

Notice that ρ = φ + ψ is the unique decomposition where the marginal φ_{2} is diffuse and the marginal ψ_{2} is purely atomic. Therefore, given two consistent parameter measures ρ_{12} and ρ_{23} on X × Y and Y × Z respectively, they can be uniquely decomposed as

ρ_{12} = φ_{12} + ψ_{12}, ρ_{23} = φ_{23} + ψ_{23},

where φ_{12} and φ_{23} are consistent, with diffuse marginal φ_{2}, whereas ψ_{12} and ψ_{23} are consistent, with purely atomic marginal ψ_{2}.

Assume from now on that X, Y and Z are Borel spaces. Then the previous corollary allows to construct a general hyper Dirichlet process, i.e. a process whose law is the hyper Markov combination of two hyperconsistent laws DI(ρ_{12}) and DI(ρ_{23}) on X × Y and Y × Z, respectively. The measures φ_{2} and ψ_{2} being singular, the Markov combination ρ of ρ_{12} and ρ_{23}, as defined in (11), is the sum of the Markov combination φ of φ_{12} and of φ_{23} and the Markov combination ψ of ψ_{12} and ψ_{23}. Let us define

ρ_{2}^{(0)}(dy) = φ_{2}(dy), ρ_{2}^{(l)}(dy) = ψ_{2}({y_{l}})δ_{y_{l}}(dy), l = 1, 2, ...

so that ρ_{2} = φ_{2} + ψ_{2} = ∑_{l=0}^{∞} ρ_{2}^{(l)}. Now let R be a sequence of independent random variables distributed as in (41), and let S = T(R), so that

S ∼ DI(ρ_{2}^{(l)}(Y), l = 0, 1, 2, ...)  (42)

Moreover let Γ ∼ HDI(φ_{12}, φ_{23}), and Ξ_{l}^{x} and Ξ_{l}^{z} be Dirichlet processes on X and Z respectively, with parameter measures ψ_{12}^{(l)}(dx) = ψ_{12}(dx × {y_{l}}) and ψ_{23}^{(l)}(dz) = ψ_{23}({y_{l}} × dz), all mutually independent. Notice that hyperconsistency implies that

ψ_{12}^{(l)}(X) = ψ_{23}^{(l)}(Z) = ψ_{2}({y_{l}}).

Then define

Σ(dx, dy, dz) = S_{0}Γ(dx, dy, dz) + ∑_{l} S_{l}Ξ_{l}^{x}(dx)Ξ_{l}^{z}(dz)δ_{y_{l}}(dy).  (43)

The following generalization of Theorem 2 holds true.

Proposition 9 The law of Σ defined in (43) is the hyper Markov combination of DI(ρ_{12}) and DI(ρ_{23}).

Proof. By using the previous Corollary the laws of the marginals Σ_{12} and Σ_{23} are immediately identified to be DI(ρ_{12}) and DI(ρ_{23}). Next, independently of S, use independent Sethuraman representations (13)

Γ = κ(γ, (U, V , W))
Ξ_{l}^{x} = κ(γ_{l}^{x}, U_{l}^{x}),  Ξ_{l}^{z} = κ(γ_{l}^{z}, W _{l}^{z}), l = 1, 2, …,

where γ, U, V , W are as in Proposition 3, γ_{l}^{x} = (γ_{l,m}^{x}, m = 1, 2, ...) and γ_{l}^{z} = (γ_{l,m}^{z}, m = 1, 2, ...) are independent sequences obtained as the image under T of i.i.d. sequences

β_{l}^{x} = (β_{l,m}^{x} ∼ Beta(1, ψ_{2}({y_{l}})), m = 1, 2, ...),
β_{l}^{z} = (β_{l,m}^{z} ∼ Beta(1, ψ_{2}({y_{l}})), m = 1, 2, ...),

respectively, and

U_{l}^{x} = (U_{l,m}^{x} ∼ ψ_{12}(· × {y_{l}})/ψ_{2}({y_{l}}), m = 1, 2, ...)
W _{l}^{z} = (U_{l,m}^{z} ∼ ψ_{23}({y_{l}} × ·)/ψ_{2}({y_{l}}), m = 1, 2, ...)

are sequences of i.i.d. random variables.

With probability 1 the sequence V contains neither repetitions nor any of the y_{l}, for l = 1, 2, . . .. This implies that the realizations of Σ are Markovian with probability 1.

As for the hyper Markov property observe that

σ {Σ_{2}} ⊂ σ {S, γ, V} ,
σ {Σ_{12}} ⊂ σ {S, γ, U, V, γ_{l}^{x}, U_{l}^{x}, l = 1, 2, … } ,
σ {Σ_{23}} ⊂ σ {S, γ, V, W, γ_{l}^{z}, W _{l}^{z}, l = 1, 2, … } ,

so that, by the conditional independence of U and W given S, γ and V , and the independence of γ_{l}^{x}, U_{l}^{x}, γ_{l}^{z}, W_{l}^{z}, l = 1, 2, … from all the remaining sequences, it is immediately obtained that

Σ_{12} ⊥ Σ_{23} | S, γ, V.

The proof is finished as in Proposition 3, the only change being that

E [ m ; ∏_{l=1} (Σ_{12}(F_{l}))^{d_{l}} | σ {S, γ, V }]

= ∫_{F_{l_{1},Y}} ⋯ ∫_{F_{l_{d},Y}} ψ̃(y_{1}, ⋯, y_{d}) Σ_{2}(dy_{1}) ⋯ Σ_{2}(dy_{d}),  (44)

where d = d_{1} + ⋯ + d_{m}, and where ψ̃ is the bounded measurable function

ψ̃(v) = E [δ_{U_{1}} ( ⋂_{i∈A_{1}(v)} F_{l_{i},X}^{(v′_{1})}) |V_{1} = v′_{1}] … E [δ_{U_{f}} ( ⋂_{i∈A_{f} (v)} F_{l_{i},X}^{(v′_{f})}) |V_{1} = v′_{f} ]

… E [ ∏_{i∈B_{1}(v)}Ξ_{j_{1}}^{x} (F_{l_{i},X}^{(y_{j_{1}})})] … E [ ∏_{i∈B_{e}(v)}Ξ_{j_{e}}^{x} (F_{l_{i},X}^{(y_{je})})] ,

with v = (v_{1}, … v_{d}), and

{v_{1}, …, v_{d}} = {v′_{1}, …, v′_{f}, y_{j_{1}}, …, y_{j_{e}}} ,

A_{1}(v), …, A_{f}(v) being defined as in (18), and

B_{1}(v) = {i : v_{i} = y_{j_{1}}}, …, B_{e}(v) = {i : v_{i} = y_{j_{e}}}.

We call a process having the same law as Σ defined in (43) a hyper Dirichlet process, extending the previously given definition. This law is again denoted by HDI(ρ_{12}, ρ_{23}). The reader will then expect that in this more general case the hyper Dirichlet process is not necessarily a Dirichlet process. In fact the following result is easily established.

Proposition 10 The process Σ defined in (43) is a Dirichlet process if and only if for each l = 1, 2, ... either ψ_{12}^{(l)} or ψ_{23}^{(l)} is a single mass. If this happens the parameter measure of Σ is the Markov combination of ρ_{12} and ρ_{23}.

Proof. If ψ_{12}^{(l)} or ψ_{23}^{(l)} is a single mass it is clear that Ξ_{l}^{x}(dx)Ξ_{l}^{z}(dz)δ_{y_{l}}(dy) is a Dirichlet process with parameter measure having total mass ψ_{2}({y_{l}}). On the other hand Γ is always a Dirichlet process with parameter measure having total mass φ_{2}(Y). Hence, by Proposition 7, the sufficiency of the condition is established.

The necessity of the condition is obtained from the observation that, by (4) of Proposition 1, for l = 1, 2, ⋯

\frac{Σ(dx × \{y_{l}\} × dz)}{Σ(X × \{y_{l}\} × Z)} = Ξ_{l}^{x}(dx)Ξ_{l}^{z}(dz)

is a Dirichlet process. Conversely, by taking any finite partition {B_{i}} of X and {D_{k}} of Z, and arguing as in the proof of Proposition 2 for the finite case, the necessity follows.

Next our aim is to establish the form of the law of a hyper Dirichlet process Σ conditional to a random sample (X_{i}, Y_{i}, Z_{i}), i = 1, .., n, drawn from Σ itself. This information is not supplied by the general conjugacy result proved in Dawid and Lauritzen (Corollary 5.5 of [3]), which holds only for strong hyper Markov processes. We begin by constructing a single observation (X, Y, Z) = (X_{1}, Y_{1}, Z_{1}) in the same space where Σ is constructed. This is achieved by using additional label variables.

Consider first a random probability distribution on some measurable space (E, E) of the form

H = +∞ ; ∑_{l=0}S_{l}Γ_{l},  (45)

where Γ = (Γ_{l}, l ≥ 0) is a sequence of independent random probability distributions on (E, E) with Γ_{l} ∼ η_{l}, and S = (S_{l}, l ≥ 0) is a random probability distribution on ℕ independent of Γ, with law ν.

Conditionally to (S, Γ), let N be an integer random variable with law (S_{l}, l ≥ 0), and let ξ have law Γ_{N}, conditionally to (S, Γ) and N. In this way, conditionally to σ(H), ξ has law H (see Lemma 4.1 in[10]), thus we say that ξ is a sample from H.

Lemma 11 Suppose there exists a family {A_{l}, l ≥ 0} of disjoint subsets of E such that Γ_{l}(A_{l}) = 1 with probability 1. Then

L (S, Γ|ξ) = L (S|N = n) ⊗_{l≠n} L (Γ_{l}) ⊗ L (Γ_{n}|ξ) , if ξ ∈ A_{n}.  (46)

Proof. The condition on the partition implies that the representation (45) is unique so we may identify (S, Γ) with H, and moreover

N = ∑_{l=0}^{∞} l 1_{A_{l}}(ξ),  (47)

Hence

L (S, Γ|ξ) = L (S, Γ|N = n, ξ) , if ξ ∈ A_{n},  (48)

Since by assumption

P (S ∈ ds, N = n, Γ ∈dg, ξ ∈ dx) = ν(ds)s_{n}∏_{l}η_{l}(dg_{l})g_{n}(dx),

it is

P (S ∈ ds, Γ ∈dg, ξ ∈ dx|N = n) = \frac{ν(ds)s_{n}}{∫ ν(dq)q_{n}} · ∏_{l≠n}η_{l}(dg_{l}) · η_{n}(dg_{n})g_{n}(dx),

which ensures that S, {Γ_{l}, l ≠ n} and (Γ_{n}, ξ) are conditionally independent given N = n, which implies (46).

This observation is the key to establish the following result.

Proposition 12 Consider a hyper Dirichlet process Σ ∼ HDI (ρ_{12}, ρ_{23}). Assume that, conditionally on Σ, the vector of observations

((X_{i}, Y_{i}, Z_{i}), i = 1, ..., m)

has i.i.d. components, each with distribution Σ. Then, conditionally on the observations, the law of Σ is HDI (ρ_{12} + ∑_{i=1}^{m} δ_{(X_{i},Y_{i})}, ρ_{23} + ∑_{i=1}^{m} δ_{(Y_{i},Z_{i})}) .

Proof. Let us consider the case of a sample of size m = 1. For larger values of m the result is obtained through the standard recursion argument for constructing posterior distributions (see e.g. Sethuraman [10], page 648). The hyper Dirichlet process Σ defined in (43) has the form (45) with Γ_{0}(dx, dy, dz) = Γ(dx, dy, dz) and Γ_{l}(dx, dy, dz) = Ξ_{l}^{x}(dx)Ξ_{l}^{z}(dz)δ_{y_{l}}(dy), l = 1, 2, ..., and in our case ξ = (X, Y, Z). Moreover A_{0} = {(x, y, z) ∈ X × Y × Z : y ≠ y_{l}, l = 1, 2, ...} and A_{l} = X × {y_{l}} × Z, l = 1, 2, ...

Since S ∼DI(ρ_{2}^{(l)}(Y), l ∈ ℕ) and N is drawn from S, the standard result about conjugacy of Dirichlet processes ([10], Lemma 4.1) yields that S is DI(ρ_{2}^{(l)}(Y) + δ_{n}(l), l ∈ ℕ), conditional to N = n. Next, for the same reason, the law of Γ conditional to (X, Y, Z) (for Y ∉ {y_{l}, l = 1, 2, ...}) is DI(ρ + δ_{(X,Y,Z)}), where ρ is the Markov combination of ρ_{12} and ρ_{23} . It is easily seen that ρ + δ_{(X,Y,Z)} is the Markov combination of ρ_{12} + δ_{(X,Y)} and ρ_{23} + δ_{(Y,Z)}, thus

DI(ρ + δ_{(X,Y,Z)}) = HDI(ρ_{12} + δ_{(X,Y)}, ρ_{23} + δ_{(Y,Z)}).

Finally, when Y = y_{l}, the law of the independent Dirichlet processes Ξ_{l}^{x} and Ξ_{l}^{z} given samples X and Z extracted independently from each of them is the product law of two Dirichlet processes ψ_{12}^{(l)} + δ_{X} and ψ_{23}^{(l)} + δ_{Z}. Putting together all these results in (46) and using Proposition 9 with ρ_{12} + δ_{(X,Y)} and ρ_{23} + δ_{(X,Y,Z)} replacing ρ_{12} and ρ_{23} the statement of the proposition for m = 1 is immediately obtained.

Remark 13 The previous theorem states that the class of hyper Dirichlet processes is conjugate w.r.t. random sampling, as it happens for the class of Dirichlet processes. Now observe that the law of a hyper Dirichlet process conditional to the observations has always parameter measures with atomic components. But if the prior law is HDI(ρ_{12}, ρ_{23}) with ρ_{2} = φ_{2} diffuse, the probability of observing (Y_{i}, i = 1, ..., m) with all distinct components is 1. By consequence the atomic component of the posterior parameter measure is a sum of Dirac measures, hence, by Proposition 10, the posterior law is not only the law of a hyper Dirichlet process but actually the law of a Dirichlet process, as it results directly from the conjugacy property of these latter processes.

We now finally pass to a limit result similar to Theorem 4 for general Hyper Dirichlet processes. From now on we take again X, Y and Z to be Euclidean spaces with dimensions d_{1}, d_{2} and d_{3}, respectively, and assume that for each of these spaces we have a sequence of finite partitions as in (19). Correspondingly, we discretize along the corresponding product sequences of partitions in two-dimensional cells (20) the parameter measures ρ_{12} = φ_{12} + ψ_{12} and ρ_{23} = φ_{23} + ψ_{23}, thereby getting the tables

ρ_{12}^{(n)} = φ_{12}^{(n)} + ψ_{12}^{(n)}, ρ_{23}^{(n)} = φ_{23}^{(n)} + ψ_{23}^{(n)},

where φ_{12}^{(n)} and φ_{23}^{(n)} are defined in (21), and

ψ_{12}^{(n)} = {ψ_{12}(B_{i}^{(n)} × C_{j}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}} ,
ψ_{23}^{(n)} = {ψ_{23}(C_{j}^{(n)} × D_{k}^{(n)}), j ∈ J^{(n)}, k ∈ K^{(n)}} .

For each integer n consider a HDi(ρ_{12}^{(n)}, ρ_{23}^{(n)}) distributed random array

{F^{(n)}(i, j, k), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}}  (49)

which has to be compared with

{Σ^{(n)}(i, j, k) = Σ(B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}} (50)

where Σ ∼ HDI (ρ_{12}, ρ_{23}) is defined in (43). However, for our next result it is more convenient to use a different representation for this process which is constructed next.

Let us proceed to construct both (49) and (50) on the same probability space. We start by defining Γ^{(n)} and G^{(n)} = G_{3|2}^{(n)} as in Theorem 4. All the other random variables to be defined will be taken independent of these two arrays. Let S_{0} ∼ Be(φ_{2}(Y), ψ_{2}(Y)) be the weight of the diffuse part in (43). Then we split the atomic part as a mixture of distributions, each concentrated on a set of the partition {X × C_{j}^{(n)} × Z , j ∈ J^{(n)}}. For this we need first an independent vector of masses

(M_{1}^{(n)}, ..., M_{|J^{(n)}|}^{(n)}) ∼ Di(ψ_{2}(C_{j}^{(n)}), j ∈ J^{(n)})

Next we have to split the mass inside each set of the partition. So for each j ∈ J^{(n)} such that ψ_{2}(C_{j}^{(n)}) > 0, define the following random variables. First an independent sequence of independent random variables L_{j}^{(n)} =

{L_{j,m}^{(n)}, m = 1, 2, ...} with

L_{j,m}^{(n)} ∼ Be (ψ_{2}(y_{j,m}^{(n)}), ∑_{i=m+1}^{∞} ψ_{2}(y_{j,i}^{(n)})) , m = 1, 2, ...,

where {y_{j,m}^{(n)}, m = 1, 2, ...} is the subset {y_{l}, l = 1, 2, ...} ∩ C_{j}^{(n)}, with the points ordered in an arbitrary way. The sequence L_{j}^{(n)} is used to obtain the corresponding sequence of weights U_{j}^{(n)} = T(L_{j}^{(n)}), i.e.

U_{j,m}^{(n)} = ∏_{i=1}^{m−1}(1 − L_{j,i}^{(n)})L_{j,m}^{(n)}, m = 1, 2, ...

which is a Dirichlet process on ℕ with parameter measure ∑_{m} ψ_{2}(y_{j,m}^{(n)})δ_{m}, as it results by the application of Proposition 7. Notice that in case the sequence {y_{j,m}^{(n)}, m = 1, 2, ...} is finite, L_{j}^{(n)} and U_{j}^{(n)} are both taken to be eventually zero. Observe that, by Proposition 1, the joint law of

{M_{j}^{(n)}U_{j,m}^{(n)}, m = 1, 2, ..., j ∈ J^{(n)}}

is the same as that of

\{\frac{S_{j,m}^{(n)}}{1 − S_{0}} , m = 1, 2, ..., j ∈ J^{(n)}\},

where S_{j,m}^{(n)} is defined to be S_{l} as given by (42) for the index l ≥ 1, such that y_{l} = y_{j,m}^{(n)}. Moreover both these sequences are independent of S_{0}.

Finally, for each j such that ψ_{2}(C_{j}^{(n)}) > 0, mutually independent Dirichlet processes Ξ_{j,m}^{x,(n)} and Ξ_{j,m}^{z,(n)} are defined on X and Z, with parameter measures ψ_{12}(dx × {y_{j,m}^{(n)}}) and ψ_{23}({y_{j,m}^{(n)}} × dz), m = 1, 2, ... We have thus completed the construction of the probability space (depending on n) over which we define

Σ^{(n)}(i, j, k)

= S_{0}Γ^{(n)}(i, j, k) + (1 − S_{0})M_{j}^{(n)} ∑_{m} U_{j,m}^{(n)}Ξ_{j,m}^{x,(n)}(B_{i}^{(n)})Ξ_{j,m}^{z,(n)}(D_{k}^{(n)}) (51)

As far as the approximation is concerned, we also need independent copies of the sequences L_{j}^{(n)} = {L_{j,m}^{(n)}, m = 1, 2, ...}, say L_{j}^{′(n)} = {L_{j,m}^{′(n)}, m = 1, 2, ...}, with the corresponding sequence of weights U_{j}^{′(n)} = {U_{j,m}^{′(n)}, m = 1, 2, ...} = T(L_{j}^{′(n)}), and an independent random vector with independent components O_{j}^{(n)} ∼ Be(φ_{2}(C_{j}^{(n)}), ψ_{2}(C_{j}^{(n)})), j ∈ J^{(n)}. Then we define

F^{(n)}(i, j, k) = {S_{0}Γ_{12}^{(n)}(i, j) + (1 − S_{0})M_{j}^{(n)} ∑_{m} U_{j,m}^{(n)}Ξ_{j,m}^{x,(n)}(B_{i})}
· {O_{j}^{(n)}G^{(n)}(k|j) + (1 − O_{j}^{(n)}) ∑_{p} U_{j,p}^{′(n)}Ξ_{j,p}^{z,(n)}(D_{k})} (52)

By using Proposition 1 and Proposition 7, it is easy to check that the marginal laws of the arrays Σ^{(n)} and F^{(n)} are the required ones. Moreover the following result holds.

Theorem 14 Under the assumptions ii) and iii) of Theorem 4

lim_{n→∞} ∑_{i∈I^{(n)}} ∑_{j∈J^{(n)}} ∑_{k∈K^{(n)}} E|Σ^{(n)}(i, j, k) − F^{(n)}(i, j, k)| = 0.

Proof. We subtract (52) from (51), and we bound from above its absolute value by

S_{0}|Γ^{(n)}(i, j, k) − Γ^{(n)}(i, j)O_{j}^{(n)}G^{(n)}(k|j)|  (53)

+ (1 − S_{0})M_{j}^{(n)} ∑_{m} U_{j,m}^{(n)}Ξ_{j,m}^{x}(B_{i}) |Ξ_{j,m}^{z,(n)}(D_{k}) − (1 − O_{j}^{(n)}) ∑_{p} U_{j,p}^{′(n)}Ξ_{j,p}^{z,(n)}(D_{k})| (54)

+ S_{0}Γ^{(n)}(i, j)(1 − O_{j}^{(n)}) ∑_{p} U_{j,p}^{′(n)}Ξ_{j,p}^{z,(n)}(D_{k})  (55)
+ (1 − S_{0})M_{j}^{(n)}O_{j}^{(n)}G^{(n)}(k|j) ∑_{m} U_{j,m}^{(n)}Ξ_{j,m}^{x,(n)}(B_{i}).  (56)

We then proceed to bound each of the four terms. For the first term (53) notice that

S_{0}|Γ^{(n)}(i, j, k) − Γ^{(n)}(i, j)O_{j}^{(n)}G^{(n)}(k|j)|

≤ S_{0}|Γ^{(n)}(i, j, k) − Γ^{(n)}(i, j)G^{(n)}(k|j)| + S_{0}Γ^{(n)}(i, j)(1 − O_{j}^{(n)})G^{(n)}(k|j)

Since Γ^{(n)}(i, j)G^{(n)}(k|j) = G^{(n)}(i, j, k), by summing over i, j, k and taking the expected value, the first term goes to zero in the mean by Theorem 4. For the second term, by summing and taking the mean we get

E [S_{0}] ∑_{i,j,k} E [Γ^{(n)}(i, j)] E [1 − O_{j}^{(n)}] E [G^{(n)}(k|j)]

≤ \frac{φ_{2}(Y)}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{i,j} \frac{φ_{12}(B_{i}^{(n)} × C_{j}^{(n)})}{φ_{12}(X × Y)} \frac{ψ_{2}(C_{j}^{(n)})}{φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)})}

= \frac{1}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} \frac{φ_{2}(C_{j}^{(n)})ψ_{2}(C_{j}^{(n)})}{φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)})} =: χ_{n}.

In order to deal with the second term (54) observe that

|Ξ_{j,m}^{z,(n)}(D_{k}) − (1 − O_{j}^{(n)}) ∑_{p} U_{j,p}^{′(n)}Ξ_{j,p}^{z,(n)}(D_{k})|

≤ (1 − O_{j}^{(n)}) ∑_{p} U_{j,p}^{′(n)} |Ξ_{j,m}^{z,(n)}(D_{k}) − Ξ_{j,p}^{z,(n)}(D_{k})| + O_{j}^{(n)}Ξ_{j,m}^{z,(n)}(D_{k}).

Then, inserting the above estimate in (54), summing both sides over i, j, k, and taking the expected value we get the upper bound

E(1 − S_{0})·

· { ∑_{j} EM_{j}^{(n)}E(1 − O_{j}^{(n)}) ∑_{m} ∑_{p≠m} EU_{j,m}^{(n)}EU_{j,p}^{′(n)} ∑_{k} E |Ξ_{j,m}^{z,(n)}(D_{k}) − Ξ_{j,p}^{z,(n)}(D_{k})|

+ ∑_{j} EM_{j}^{(n)}EO_{j}^{(n)}}  (57)

The last term in (57) is the easiest to deal with, since it is equal to

E(1 − S_{0}) ∑_{j} EM_{j}^{(n)}EO_{j}^{(n)}

= \frac{ψ_{2}(Y)}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} \frac{ψ_{2}(C_{j}^{(n)})}{ψ_{2}(Y)} \frac{φ_{2}(C_{j}^{(n)})}{φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)})} = χ_{n},

whereas the first term in (57) is bounded by

2E(1 − S_{0}) ∑_{j} EM_{j}^{(n)}E(1 − O_{j}^{(n)}) ∑_{m≠p} ∑ EU_{j,m}^{(n)}EU_{j,p}^{′(n)}

= 2\frac{ψ_{2}(Y)}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} \frac{ψ_{2}(C_{j}^{(n)})^{2}}{ψ_{2}(Y) (φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)}))} ∑_{m} ∑_{p≠m} \frac{ψ_{2}(\{y_{j,m}^{(n)}\})ψ_{2}(\{y_{j,p}^{(n)}\})}{ψ_{2}(C_{j}^{(n)})^{2}}

≤ \frac{4}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} \frac{ψ_{2}(C_{j}^{(n)})}{φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)})} ∑_{p>1} ψ_{2}(\{y_{j,p}^{(n)}\})

≤ \frac{4}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} ∑_{p>1} ψ_{2}(\{y_{j,p}^{(n)}\}) =: τ_{n},

where we have used the fact that, for any sequence a_{n} ≥ 0

∑_{n} ∑_{m≠n}a_{n}a_{m} = 2 ∑_{n} ∑_{m>n}a_{n}a_{m} ≤ ∑_{n} a_{n}∑_{m>1}a_{m}.

Likewise, the expected value of the sum of the third terms (55) yields

ES_{0} ∑_{j} E(∑_{i} Γ^{(n)}(i, j))E(1 − O_{j}^{(n)})

= \frac{φ_{2}(Y)}{φ_{2}(Y) + ψ_{2}(Y)} ∑_{j} \frac{φ_{2}(C_{j}^{(n)})}{φ_{2}(Y)} \frac{ψ_{2}(C_{j}^{(n)})}{φ_{2}(C_{j}^{(n)}) + ψ_{2}(C_{j}^{(n)})} = χ_{n},

and for the last terms (56) we bound the sum of the expected values by

E(1 − S_{0}) ∑_{j} EM_{j}^{(n)}EO_{j}^{(n)} = χ_{n}.

We finally end the proof by noticing that, since ψ_{2} is a finite measure, for any ϵ > 0 there exists an integer L_{ϵ} such that

∑_{l=1}^{L_{ϵ}} ψ_{2}({y_{l}}) ≥ ψ_{2}(Y) − ϵ;

moreover, for n large enough each of the C_{j}^{(n)} will contain either one or none of the y_{l}, l = 1, 2, ..., L_{ϵ}. Therefore, for n large enough, ∑_{j} ∑_{p>1} ψ_{2}({y_{j,p}^{(n)}}) ≤ ϵ, so that

τ_{n} ≤ \frac{4ϵ}{φ_{2}(Y) + ψ_{2}(Y)}

and analogously, for n large enough,

χ_{n} ≤ \frac{1}{φ_{2}(Y) + ψ_{2}(Y)} (ϵ + L_{ϵ} max_{j∈J^{(n)}} φ_{2}(C_{j}^{(n)})) ≤ \frac{2ϵ}{φ_{2}(Y) + ψ_{2}(Y)}

since max_{j∈J^{(n)}} φ_{2}(C_{j}^{(n)}) = λ_{n} goes to zero as n → ∞, as it has been proved in Theorem 4. Thus the proof is finished.

As a corollary of the above result we get immediately the announced generalization of Theorem 5.

Theorem 15 Under the assumption of the previous theorem, let F^{(n)} be any random probability distribution on X × Y ×Z = ℝ^{d}, with the property that

{F^{(n)}(B_{i}^{(n)} × C_{j}^{(n)} × D_{k}^{(n)}), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}}

has the same law as

{F^{(n)}(i, j, k), i ∈ I^{(n)}, j ∈ J^{(n)}, k ∈ K^{(n)}} .

Furthermore suppose that (32) and (33) hold for all compact sets E ⊂ X and F ⊂ Z. Then {F^{(n)}} converges weakly to Σ, as probability measures on ℝ^{d}, endowed with the topology of weak convergence.

References

[1] Billingsley, P. Convergence of probability measures, second ed. Wiley Series in Probability and Statistics: Probability and Statistics. John Wiley & Sons Inc., New York, 1999. A Wiley-Interscience Publication.

[2] Blackwell, D., and MacQueen, J. B. Ferguson distributions via Pólya urn schemes. Ann. Statist. 1 (1973), 353–355.

[3] Dawid, A. P., and Lauritzen, S. L. Hyper-Markov laws in the statistical analysis of decomposable graphical models. Ann. Statist. 21, 3 (1993), 1272–1317.

[4] Dubins, L., and Freedman, D. Measurable sets of measures. Pacific J. Math. 14 (1964), 1211–1222.

[5] Ferguson, T. S. A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 (1973), 209–230.

[6] Kotz, S., Balakrishnan, N., and Johnson, N. L. Continuous multivariate distributions. Vol. 1, second ed. Wiley Series in Probability and Statistics: Applied Probability and Statistics. Wiley-Interscience, New York, 2000. Models and applications.

[7] Lauritzen, S. L. Graphical models, vol. 17 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 1996. Oxford Science Publications.

[8] Lindvall, T. Lectures on the coupling method. Dover Publications Inc., Mineola, NY, 2002. Corrected reprint of the 1992 original.

[9] Rachev, S. T. Probability metrics and the stability of stochastic models. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. John Wiley & Sons Ltd., Chichester, 1991.

[10] Sethuraman, J. A constructive definition of Dirichlet priors. Statist. Sinica 4, 2 (1994), 639–650.

[11] Sethuraman, J., and Tiwari, R. C. Convergence of Dirichlet measures and the interpretation of their parameter. In Statistical decision theory and related topics, III, Vol. 2 (West Lafayette, Ind., 1981). Academic Press, New York, 1982, pp. 305–315.