Introduction
This series of webpages is designed to provide you with an explanation of the basic concepts of information collecting and processing systems. To do this we will examine examples ranging from secret codes to compact disc players. Using these practical examples you should be able to see how the mathematics of Information Theory can be applied in practical situations to make Instruments which perform useful tasks. This first part is intended to a be a general outline. Most of the concepts introduced here will be looked at more carefully later.
Scientists and engineers devote considerable attention to the processing and storage of information, yet questions relating to how information is produced generally attract less consideration. To some extent, this blind spot seems to stem from a belief that any interest in this area smells strongly of philosophy, not engineering. In general, practically minded scientists don't want to ‘waste their time’ with philosophy — although there are many notable exceptions to this rule.
These pages are not about philosophy. No time will be devoted to questions like:
‘What is the meaning of meaning?’
‘How do we know what we know?’
etc.
Despite this, when trying to understand information based systems it's vital to have some idea of how information is created or captured.
What is information?
For our purposes, we can say that information initially comes from some form of sensor or transducer. This generates some form of response which can then be measured. It is this measurement or detection which ‘creates’ information. (In fact, the sensor is reacting to the arrival of some input pattern of energy or power. It would be fairer to say it ‘picks up’ the information, but we'll ignore this fact.) Once we adopt this starting point it becomes clear that the topics of instrumentation and measurement form the basis of all practical information systems.
This viewpoint provides us with a double advantage over someone who is studying information theory purely as a branch of mathematics. Firstly, it gives us a way to understand information processing systems in terms of the physical properties of the real world. Secondly, it helps us sort out questions related to the ‘value’ or ‘meaning’ of information without the risk of being dragged into metaphysics. Instead we can simply ask, ‘How was this information produced?’
What is an ‘instrument’? At first glance, it can appear to science and engineering students that the subject called Instrumentation is obsessed with describing how voltmeters and oscilloscopes work. Yet the subject covers a much wider and more important area. A colour TV is an instrument. A digital computer is an instrument. Each senses some form of input and responds by producing an appropriate output. The TV responds to an electromagnetic wave from a distant transmitter to produce a corresponding picture on a screen and sound from a loudspeaker. The computer can be affected by various sorts of input, from a keyboard, a mouse, or by reading a magnetic disc. It can respond by altering the electronic pattern held in its memory, by altering its monitor display, or recording something on a disc.
Most of the examples we'll look at in these pages will be electronic or optical. This is because optical and electronic methods are powerful and widely used. Despite this, it's important to realise that the basic points made here aren't only true in these areas. To emphasise this, we can start by considering a simple mechanical measurement system — a kitchen balance — to make some fundamental points which apply to all measurement (information gathering) systems.
The balance has a pan or plate supported by a spring. When we place something on the pan the added weight presses down on the spring, compressing it. The pan moves downwards until the compression force from the squeezed spring balances the force of the increased weight. Most balances have a rotary dial with a pointer attached to the pan. The movement caused by the weight rotates the pointer to give us a ‘reading’ of the weight.
The first point to note is that, like most measurement systems, this one is indirect. What we actually observe is a movement (rotation) of the pointer. We don't actually see the magnitude of the weight. If, for example, we put an iron on the pan we might see the pointer move around through 120 degrees. If we liked, we could also use a ruler to find that the pan moved down 2 cm. However, we don't usually quote weights in degrees or centimetres! In order to make sense of these observed values we have to calibrate the balance. To do this we can place two or three different known weights on the scales and make a note of how far the pointer goes around (or the pan falls) each time. We can then use these results to make a series of calibration marks on the face of the dial. Now, when we put something — e.g an iron — on the scales we can read off its weight from the dial. This calibration process means that the balance provides us with a means to compare the weight of the iron with a set of other ‘standard’ weights. In general, all measurements are Comparisons with some defined standard.
Usually, we buy a kitchen balance which should already be calibrated (i.e. its dial is marked in kg, lb, etc, not degrees) and we don't bother to calibrate the weighing instrument for ourselves. However, when we consider the need for a calibration process an awkward question springs to mind — where did the ‘known’ weights come from that were used to calibrate the readings? If all measurements are comparisons, how were the values of those weights known? They, too, would need to have been weighed on some weight measurement system. If so, how was that system calibrated?
Any measurement we make is the last link in a chain of similar measurements. Each one calibrates a system or a ‘standard’ (e.g. a known weight) which can be used for the next step. Right back at the beginning of this chain (at the National Physical Laboratory in the UK and other standards labs around the world) there will a Primary Reference system or standard which is used to define what we mean by ‘1 kg’, or ‘1 second’ or whatever. In effect, when we plonk something on the pan of a kitchen balance we're indirectly comparing it with the standard kg weight kept under a glass cover at the NPL.
When we place an iron on the pan we have to wait a second or two to let the system settle down and allow the pointer to stop moving. Similarly, when we remove the iron the system takes a short time to recover. The second point we can make about the measurement system is, therefore, that it has a finite Response Time — i.e. we have to wait for a specific time after any change in the weight before we can make a reliable reading. This limits our ability to measure any changes which take place too quickly for the system.
The third point to note is fairly obvious from our choice of an iron. If we put too large a weight on the pan the pointer will go right around and move ‘off scale’. (If the iron is very heavy we may even smash the scales!) No matter how well we search the shops, we can't find scales which can accurately measure any weight, no matter how big. Every real instrument is limited to operate over some finite Range. Beyond this range it won't work properly and Overloads or Saturates to give a meaningless response.
The fourth and final basic point is something we won't usually notice using ordinary scales since the effect is relatively small. All of the atoms in the scales, including those in its spring, will be at room temperature. (In a kitchen this probably means at or above 20 Celsius or 293 Kelvin.) As a result, they'll be moving around with random thermal motions. Compared to the effect of placing an iron on the scales these movements are quite small. However, if we looked at the pointer very carefully with a powerful microscope we'd see its angle fluctuating randomly up and down a little bit because of the motions of the atoms in the spring. As a result, if we wanted to measure the weight very accurately this thermal jittering would limit the precision of our reading. As a result, no matter how good the scales, our ability to make extremely accurate measurements is limited by thermal random effects or thermal noise.
Content and pages maintained by: Jim Lesurf (jcgl@st-and.ac.uk)
using HTMLEdit and TechWriter on a StrongARM powered RISCOS machine.
University of St. Andrews, St Andrews, Fife KY16 9SS, Scotland.
