Property of the conditional entropy

The conditional entropy can be determined through a syllable entropy:

, (3.5)

. (3.6)

The conditional entropy has the following property: if the symbols and also are independent,

. (10.16)

Hence,

. (3.7)

Only for sources with independent symbols

, (3.8)

. (3.9)

10.2.3. Two reasons for decreasing of the average information quantity in the real message.

As

, (3.10)

there are two reasons for decreasing of the average information quantity in the real message:

§ non-uniformity of distribution of probabilities of symbols occurrence,

§ correlation connections between them.

In the real messages the correlation connections exist not only between two adjacentsymbols, but also between by l+1 symbols. Speak, that the source has memory on l of symbols. The formulas (5), (12), (13) are deduced for a case l = 1. Similarly define conditional entropy for common cases, when l > 1. For real sources entropy and productivity differ from maximal meanings, and the redundancy of the messages is great.

Дата добавления: 2015-05-10