Semantical Systems: Probability and Induction

          In his article "Testability and Meaning" in Philosophy of Science (1936) Carnap abandoned the idea of verification, because he concluded that hypotheses about unobserved events in the physical world can never be completely verified by observational evidence.  Thus he proposed instead the probabilistic idea of confirmation.  He became interested in the philosophy of probability in 1941, when he considered that the concept of logical probability might supply an exact quantitative explication of the concept of confirmation of a hypothesis with respect to a given body of evidence, such that it would become possible to speak of a degree of confirmation in a measurable sense.  Up to that time there were fundamentally two kinds of concepts of probability, which were proposed by their advocates as alternatives.  The earlier view is the frequency concept advanced by Richard von Mises and Hans Reichenbach.  The other view is the logical concept advanced by John Maynard Keynes in 1921 and by Harold Jeffreys in 1939, and also considered by Ludwig Wittgenstein in his Tractatus, where he defined probability on the basis of the logical ranges of propositions.  Wittgenstein's interpretation construes a probability statement to be analytic unlike the frequency concept, which construes it to be synthetic or factual.  Carnap believed that the logical concept of probability is the basis for all inductive inference, and therefore he identifies the concept of logical probability with the concept of inductive proba­bility.
          In 1950 Carnap published Logical Foundations of Proba­bility. This work on probability is not a development in the calculus of probability or in the techniques of statistical inference.  It is Carnap’s contribution to the interpretation of probability theory with the constructionalist approach, a further development of his metatheory of semantical systems.  Here his distinction between object language and metalanguage serves as the basis for his relating the con­cepts of logical and statistical probability.  Statements of statistical probability occur in an object language and are empirical statements about the world.  Statements of logical probability occur in the metalanguage and are about the degree of confirmation of statements in the object language.  Carnap also refers to the statements in the metalanguage for scientific theory as "metascientific" statements.  However, for Carnap metascientific statements are not empirical, but rather are analytic or L-true; he does not recognize an empirical metascience.  He accepts the frequency interpreta­tion for the statistical probability asserted by statements in the object language; statistical probability therefore is the relative frequency of an occurrence of an event in the long run.  Logical probability is the estimate of statisti­cal probability, and it is the measure of the degree of con­firmation.  Symbolically he expresses this logical probability as:

                    c(h,e) = r

which means that hypothesis h is confirmed by evidence e to the degree r.  The variable r is the measure of the degree of confirmation, such that r can take values from 0.0 to 1.0; it is the estimate of the relative frequency and is expressed as:

                    r = m(e*h)/m(e)

where m(e*h) is the number of observation sentences descri­bing observed confirming instances e of hypothesis h, and m(e) is the number of observation sentences e describing the total number of observed instances, both confirming and disconfirming.  He calls m a measurement function.
          In Carnap's view the logical foundation of probability is logic in the sense of L-truth, and he therefore draws upon his metatheory of semantical systems, in which his ideas of state description and range have a central role.  A state description is a conjunction containing for every atomic sentence that can be formed in a language, either its affirmation or its negation but not both.  Thus every L-true sen­tence is true in all the state descriptions, and every L-false or self-contradictory sentence is false in every state description.  The F-true or factually true sentences are true in only some state descriptions but are not true in others.  When the idea of state description is related to the concept of logical probability, the L-true sentences have a degree of confirmation of 1.0, and the L-false sentences have a degree of confirmation of 0.0.  The F-true sentences on the other hand have a degree of confirmation between 1.0 and 0.0.  A closely related concept is that of the range of a statement.  The range is defined as the class of all state descriptions in which an empirical state­ment is true, and it may also be defined as those state descriptions that L-imply the statement.  Using the concept of range the equation r = m(e*h)/m(e) may be said to be the partial inclusion of the range of e in h as measured by r.   Therefore the equation c(h,e) = r is analogous to the statement that e L-implies h except that the range of e is not com­pletely contained in h.  Both types of statements are analy­tical or L-true statements in the metalanguage, because both are statements in logic, one in inductive logic and the other in deductive logic.  In Carnap's philosophy the logical foundations of probability is logic in the sense of L-truth.
          In 1952 Carnap published The Continuum of Inductive Methods, which was to be the volume on the theory of induc­tion that followed Logical Foundations of Probability, but he became dissatisfied with this treatment.  For many years he continued to work on induction.  At the time of his death in 1970 he had completed "Inductive Logic and Rational Decisions" and "A Basic System of Inductive Logic, Part I", which were published in Studies in Inductive Logic and Prob­ability, Volume I (ed. Carnap and Jeffrey, 1971).  Carnap did not complete Part II of "A Basic System", and it was edited for publication in 1980 by Jeffrey in Studies in Inductive Logic and Probability, Volume II.  In "Inductive Logic and Rational Decisions" Carnap is concerned with Bayesian decision theory.  In this context the term "probability" does not mean relative frequency, but rather means degree of belief.  He distinguishes the psycho­logical concept of actual degree of belief from the logical concept of rational degree of belief.  The former is empir­ical and descriptive, while the latter is normative for rational decision making.  Carnap considers the former to be subjective, since it differs from one individual person to another, while the latter is objective.  Carnap maintains that contrary to prevailing opinion relative frequency is not the only kind of objective probability.  He also calls the former "actual credence" and the latter "rational credence.”  Rational credence is the link between descriptive theory and inductive logic, and like inductive logic it is formal, deductive and axiomatic.  The concepts of inductive logic and of normative decision theory are similar but not identical.  The latter are quasi psychological, while the former have nothing to do with observers and agents, even as these are generalized so that the decision theory is not subjective.  Hence there are separate measure functions and confirmation functions for rational decision theory and for inductive logic.  In his "A Basic System of Inductive Logic" Carnap develops a set-theoretic axiomatic system, which uses set connectives instead of sentence connectives, and which is equivalent to the customary axiom systems for conditional probability.

Semantical Systems: Information Theory

          In 1953 Carnap and Yehousha Bar-Hillel, professor of logic and philosophy of science at the Hebrew University of Jerusalem, Israel, jointly published "Semantic Information" in the British Journal for the Philosophy of Science.  A more elaborate statement of the theory may be found in chapters fifteen through seventeen of Bar-Hillel's Language and Information (1964).  This semantical theory of informa­tion is based on Carnap's Logical Foundations of Probability and on Shannon's theory of communication.  In the introduc­tory chapter of his Language and Information Bar-Hillel states that Carnap's Logical Syntax of Language was the most influential book he had ever read in his life, and that he regards Carnap to be one of the greatest philosophers of all time.  In 1951 Bar-Hillel received a research associateship in the Research Laboratory of Electronics at the Massachu­setts Institute of Technology.  At the time he took occasion to visit Carnap at the Princeton Institute for Advanced Study.  In his "Introduction" to Studies in Inductive Logic and Probability, Volume I, Carnap states that during this time he told Bar-Hillel about his ideas on a semantical con­cept of content measure or amount of information based on the logical concept of probability.  This is an alternative concept to Shannon's statistical concept of the amount of information.  Carnap notes that frequently there is confusion between these two concepts, and that while both the logical and statistical concepts are objective concepts of probability, only the second is related to the physical concept of entropy.  He also reports that he and Bar-Hillel had some discussions with John von Neumann, who asserted that the basic concepts of quantum theory are subjective and that this holds especially for entropy, since this concept is based on probability and amount of information.  Carnap states that he and Bar-­Hillel tried in vain to convince von Neumann of the exis­tence of the differences in each of these two pairs of con­cepts: objective and subjective, logical and physical.  As a result of the discussions at Princeton between Carnap and Bar-Hillel, they undertook the joint paper on semantical information.  Bar-Hillel reports that most of the paper was dictated by Carnap.  The paper was originally published as a Technical Report of the MIT Research Laboratory in 1952.
          In the opening statements of "Semantic Information" the authors observe that the measures of information developed by Claude Shannon have nothing to do with what the semantics of the symbols, but only with the frequency of their occur­rence in a transmission.  This deliberate restriction of the scope of mathematical communication theory was of great heuristic value and enabled this theory to achieve important results in a short time.  But it often turned out that impatient scientists in various fields applied the terminology and the theorems of the theory to fields in which the term "information" was used presystematically in a semantic sense.  The clarification of the semantic sense of informa­tion is very important, therefore, and in this paper Carnap and Bar-Hillel set out to exhibit a semantical theory of information that cannot be developed with the concepts of infor­mation and amount of information used by Shannon's theory.  Notably Carnap and Bar-Hillel's equation for the amount of information has a mathematical form that is very similar to that of Shannon's equation, even though the interpretations of the two similar equations are not the same.  Therefore a brief summary of Shannon's theory of information is in order at this point before further discussion of Carnap and Bar­-Hillel's theory.
          Claude E. Shannon published his "Mathematical Theory of Communication" in the Bell System Technical Journal (July and October, 1948).  The papers are reprinted together with an introduction to the subject in The Mathematical Theory of Communication (Shannon and Weaver, 1964).  Shannon states that his purpose is to address what he calls the fundamental problem of communication, namely, that of reproducing at one point either exactly or approximately a message selected at another point.  He states that the semantical aspects of communication are irrelevant to this engineering problem; the relevant aspect is the selection of the correct message by the receiver from a set of possible messages in a system that is designed to operate for all possible selections.  If the number of messages in the set of all possible messages is finite, then this number or any monotonic function of this number can be regarded as a measure of the information produced, when one message is selected from the set and with all selections being equally likely.  Shannon uses a logarith­mic measure with the base of the log serving as the unit of measure.  His paper considers the capacity of the channel through which the message is transmitted, but the discussion is focused on the properties of the source.  Of particular interest is a discrete source, which generates the message symbol by symbol, and chooses successive symbols according to probabilities.  The generation of the message is therefore a stochastic process, but even if the originator of the message is not behaving as a stochastic process, the recipi­ent must treat the transmitted signals in such a fashion.  A discrete Markov process can be used to simulate this effect, and linguists have used it to approximate an English-language message.  The approximation to English language is more successful, if the units of the transmission are words instead of letters of the alphabet.  During the years immediately following the publication of Shannon's theory linguists attempted to cre­ate constructional grammars using Markov processes.  These grammars are known as finite-state Markov process grammars.  However, after Noam Chomsky published his Syntactical Struc­tures in 1956, linguists were persuaded that natural language grammars are not finite-state grammars, but are poten­tially infinite-state grammars.
          In the Markov process there exists a finite number of possible states of the system together with a set of transition probabilities, such that for any one state there is an associated probability for every successive state to which a transition may be made.  To make a Markov process into an information source, it is necessary only to assume that a symbol is produced in the transition from one state to another.  There exists a special case called an ergodic process, in which every sequence produced by the process has the same statistical properties.  Shannon proposes a quantity that will measure how much information is produced by an information source that operates as a Markov process: given n events with each having probability p(i), then the quantity of information H is:

          In their “Semantic Information" Carnap and Bar-Hillel introduce the concepts of information content of a statement and of content element. Bar-Hillel notes that the content of a statement is what is also meant by the Scholastic adage, omnis determinatio est negatio.  It is the class of those possible states of the universe, which are excluded by the statement.  When expressed in terms of state descriptions, the content of a statement is the class of all state descriptions excluded by the state­ment.  The concept of state description had been defined previously by Carnap as a conjunction containing as compo­nents for every atomic statement in a language either the statement or its negation but not both, and no other state­ments.  The content element is the opposite in the sense that it is a disjunction instead of a conjunction.  The truth condition for the content element is therefore much less than that for the state description; in the state description all the constituent atomic statements must be true for the conjunction to be true, while for the content element only one of the constituent elements must be true for the conjunction to be true.  Therefore the content elements are the weakest possible factual statements that can be made in the object language.  The only factual state­ment that is L-implied by a content element is the content element itself.  The authors then propose an explicatum for the ordinary concept of the "information conveyed by the statement i" taken in its semantical sense: the content of a statement i, denoted cont(i), is the class of all content elements that are L-implied by the statement i.  
          The concept of the measure of information content of a statement is related to Carnap's concept of measure over the range of a statement.  Carnap's measure functions are meant to explicate the presystematic concept of logical or induc­tive probability. For every measure function a corresponding function can be defined in some way, that will measure the content of any given statement, such that the greater the logical probability of a statement, the smaller its con­tent measure.  Let m(i) be the logical probability of the statement i.  Then the quantity 1-m(i) is the measure of the content of i, which may be called the "content measure of i", denoted cont(i).  Thus:

                    cont(i) = 1- m(i).

          However, this measure does not have additivity proper­ties, because cont is not additive under inductive indepen­dence.  The cont value of a conjunction is smaller than the cont value of its components, when the two statements con­joined are not content exclusive.  Thus insisting on addi­tivity on condition of inductive independence, the authors propose another set of measures for the amount of informa­tion, which they call "information measures" for the idea of the amount of information in the statement i, denoted inf(i), and which they define as:

                    inf(i) = log  {1/[1-cont(i)]}

which by substitution transforms into:

                    inf(i) = - log m(i).

This is analogous to the amount of information in Shannon's mathematical theory of communication but with inductive probability instead of statistical probability.  They make their use of the logical concept of probability explicit when they express it as:

                    inf(h/e) = - log c(h,e)

where c(h,e) is defined as the degree of confirmation and  inf(h/e) means the amount of information in hypothesis h given evidence e.  Bar-Hillel says that cont may be regarded as a measure of the "substantial" aspect of a piece of information, while inf may be regarded as a measure of its "surprise" value or in less psychological terms of its "objective unexpectedness.”  Bar-Hillel believed that their theory of semantic information might be fruitfully applied in various fields.   However, neither Carnap nor Bar-Hillel followed up with any investigations of the applicability of their semantical con­cept of information to scientific research.  Later when Bar-Hillel’s interests turned to the analysis of natural language, he noted that linguists did not accept Carnap’s semantical views.