Why Lighting Claims Might Well Be W= rong
Paul M= archant1,*
[1]<= /a> 1Visit= ing Fellow in Statistics, Leeds Beckett University, Leeds, United Kingdom LS1 3HE
Received
16 February 2017, Accepted 02 May 2017
Abstract
This paper gives some background to cl=
aims
of benefit from road lighting and why large beneficial claims may be suspec=
t.
It places this in the context of general concerns about the unreliability of
claims in science using knowledge gained, particularly from investigations =
in
the field of health-care.
It points to the need to plan, proceed=
and
check science in unbiased and rigorous ways. It gives recommendations for m=
ore
transparency, asks for clear protocols to be produced in advance, that clear
reports are written which follow appropriate reporting guidelines and that =
the
data is accessible.
Keywords: Scientific claims, Evidence, Bias,
Statistical analysis, Public safety, Public lighting
1. Introduction
In order to know whether any action is sustainab=
le
or indeed whether any intervention works, we need good untainted evidence. =
We
need sufficiently full, unbiased, transparent information. We need to know =
what
might be wrong with the evidence presented as well as what is good about it=
.
The background to this paper is the first part o=
f my
oral contribution ¡°Does increasing road lighting increase road injuries?¡±=
to
ALAN2016, Artificial Light At Night, conference, Cluj-Napoca, Romania in
September 2016. I spoke on assessing the effect of changing road lighting in
two English cities, to white light, on injury road accidents. The work, to =
be
published, has found nothing like the tens of percent improvement due to new
lighting that has been claimed. In fact, the work suggests any reduction in
injury accidents, if it exists, is very much smaller; and indeed there is a
suggestion of increased harm. It calls into question whether the such a
relighting programme represents value for money.
Here I shall go through some issues in the
scientific process that may mislead, despite how ¡®scientific¡¯ a claim mig=
ht
appear.
For example, my two batches of comments appended=
to
the Cochrane Systematic Review ¡®Street lighting for preventing road traffic
injuries¡¯ [1] highlight some seriously problematic issues with that work. =
The
review, which contained extensive meta-analyses, claimed; ¡°The results from
this systematic review suggest that street lighting may prevent road traffic
crashes, injuries and fatalities¡±. The ¡®Plain Language Summary¡¯ and podc=
asts
are more positive about lighting benefit. The review cautioned against the
policy, widespread in the UK, of reducing street lighting at night. My crit=
ical
comments on the other hand said that nothing could be claimed because the
review was inherently seriously flawed. Perhaps the points made there expla=
in
why the ¡®LANTERNS project¡¯ to determine the eff=
ect
of reduced street lighting on road casualties and crime in England and Wale=
s,
found no detriment from implementing road lighting reductions despite invol=
ving
a large dataset from 62 local authorities [2].
2. The wider background
There is currently much concern about dubious and
irreproducible claims made in science. Much of this examination and soul
searching has taken place in the area of health-care, as in a one page Lanc=
et
editorial [3] which outlines problems discussed at a conference on the issu=
e.
As an outcome the Academy of Medical Sciences produced a number of document=
s,
of varying length and detail, attempting to address the problems identified
[4].
Concerns about the reproducibility and reliabili=
ty
of scientific claims have reached the public media, such as BBC Radio4 [5, =
6]
and the Economist magazine [7]. In January 2017, the United Kingdom (UK)
Parliament, House of Commons Select Committee on Science and Technology
launched an inquiry into research integrity to try to address the issue.
3. Transparency about Potenti=
al
Conflicts of Interest.
It is important that full disclosure of potential
conflicts of interest is given in any scientific report.
What I see to be my own potential conflicts of
interest are:
• The fact I am a lighting sceptic in that I have never seen good
evidence that new road lighting enhances public safety from crime or traffic
accidents ¡¦ (The evidence seen is not robust against a number of threats to
scientific validity, which I have written about).
• I have an interest in astronomy and I decided to look into the
matter of the claimed benefits of lighting, years ago, because light pollut=
ion
spoils astronomy.
• However I don¡¯t have financial interests.
I have long been sceptical of claims of lighting benefiting public safety [8-14] so I am clear that I do have an additional potential conflict of interest as it is often painful change one¡¯s mind. <= o:p>
However, money would seem to be a major source of
bias in research, whether that is through direct financial gain, or through
potential gain, for example via a potential job promotion or securing repeat
business for providing pleasing results to the paymaster.
The matter of investigating the specific case of
lighting, including issues around lack of transparency, is given in [12] &a=
mp;
[15].
4. Publication Bias
A huge problem is the fact that not all results =
are
published so that the evidence available is only a subset of all the
investigations that were done. This issue of ¡®publication bias¡¯ is known =
to be
still a major problem in the area of health-care. It should be remembered t=
hat
health-care research is regulated. For example, before a clinical trial tak=
es place
a detailed protocol is produced for approval, in advance of collecting any
data. It should also be publicly registered so that there is a record of it,
which helps to ensure that trials can be followed-up and do not disappear f=
rom
the scientific record.
The question must therefore arise, ¡°How much bi=
gger
is the problem in an unregulated research-area, such as investigating the
effects of lighting?¡± It would seem unlikely that the problem would be sma=
ller
and indeed it would seem likely that it might be considerably larger.
The Cochrane Handbook [16] produced by the Cochr=
ane
Collaboration, which organises systematic reviews of treatments in health-c=
are,
has material on the subject of ¡®bias¡¯. Bias is the statistical term for
assessments homing in on the wrong answer, because of systematic, as oppose=
d to
random, errors. The searchable online Handbook is a useful reference for the
issue of what makes reliable science.
A major problem is that there is a strong
possibility that studies ¡®go missing¡¯ or publishing occurs very late beca=
use of
their ¡®uninteresting¡¯ or ¡®unwelcome¡¯ findings. Such absence may be asso=
ciated
with the funder. The Handbook gives evidence for this.
That publication bias remains a problem in the
regulated world of clinical trials is recognised by the United Nations in t=
he
report of the United Nations Secretary-General¡¯s High-level Panel on acces=
s to
Medicines [17]. The recommendations section (4.3.5) concerning clinical tri=
als
on page 37 of the report are:
• Governments should require that the unidentified data on all
completed and discontinued clinical trials be made publicly available in an
easily searchable public register established and operated by existing
mechanisms such as the WHO Clinical Trials Registry Platform, clinicaltrial=
s.gov
or in peer reviewed publications, regardless of whether their results are
positive, negative, neutral or inconclusive.
• To facilitate open collaboration, reconstruction and reinvestiga=
tion
of failures, governments should require that study designs and protocols, d=
ata
sets, test results and anonymity-protected patient data be available to the
public in a timely and accessible fashion. Those undertaking clinical trials
must not prevent researchers from publishing their findings.
An example highlighting the deficiency is the
critical report [18] of the UK House of Commons Public Accounts Committee on
its investigation into the transparency and completeness of the evidence on
which hundreds of millions of UK-pounds had been spent on an anti-influenza
drug.
It was to ensure that full disclosure applies to
clinical trials that the AllTrials campaign [19] was set up under the banner
¡®All Trials Registered, All Results Reported¡¯.
Publication bias seems likely to be large in pol=
icy
¡®trials¡¯, especially when something routine like lighting is introduced a=
nd
there is no registration or protocol in advance. It is not even clear that =
all
such studies are done as experiments. It may only be after implementation t=
hat
it is decided to prepare a report; once an indication of the result becomes
available. A public registry of protocols of planned research would be a
valuable contribution to making research on lighting more reliable. Note too
that the Cochrane Handbook is critical of Non-Randomised Studies (NRS). The=
se are
the kind of studies routinely used to make claims about lighting benefit.
The protocol history for the Beyer and Ker work
itself [1] is given on page 56 of the review but there is no mention of the=
re
being any protocols of the included studies, despite the critical feedback
raising the issue of protocols.
The LANTERNS project [2] has its required protoc=
ol
available on the website of its funder, the National Institute of Health
Research, Public Health Research, See [20]. The full report of the project,=
Perkins
et al [21], is available there for examination also.
5. Statistical Issues
It must be remembered that it is not very diffic=
ult
to get a result out of a piece of statistical software. The difficult thing=
is
to make sure that a result properly addresses a real world problem, and does
not arrive at a wrong conclusion and thereby promote a false solution.
In any study through time, there is a need to
account for the underlying temporal change, because it is important not to
confuse the change which would happen due to the passage of time with that =
of
the effect of the intervention. Therefore an adequate control needs to be u=
sed.
When making a comparison of bright road lighting
with dim road lighting, it is important to ensure that the comparison made =
is fair
and ¡®like is being compared with like¡¯. The beauty of the Randomised Cont=
rol
Trial (RCT) is, that because of the random allocation of individual units to
the treatments being compared, the treatment groups are statistically
equivalent at the start. Therefore, any departure from statistical equivale=
nce
detected at the end point of the trial indicates that there has been a
difference of effect of treatments.
Beyer and Ker [1] set inclusion criteria for the=
ir
Cochrane Review. Studies were searched for as described in the review repor=
t.
The search identified 1308 relevant studies, out of which 17 met the inclus=
ion
criteria. These were all controlled before after studies; no RCTs were foun=
d.
The characteristics of studies are given in the report.
In observational studies, i.e. those without
randomisation, there may be important initial differences between the groups
being compared which may lead to a wrong conclusion.
Regression towards the Mean (RTM), also known as
Regression to the Mean and originally called Regression to Mediocrity when =
it
was first discover by Francis Galton, is one source leading to a biased
estimate of effect in studies [22]. It would be wrong to run a ¡®controlled=
¡¯
before and after study to estimate the effect of increased lighting if this
compared areas with a higher crime-rate and given an increase in lighting w=
ith
other ¡®control¡¯ areas with lower crime rates and unchanging lighting. The
biasing effect of RTM increases with the size of starting difference between
the groups being compared. It is also greater in a situation where the
correlation between the baseline and outcome measurements is weaker and it
depends on the form of the bivariate distribution. The likely size of the
biasing effect may be estimated [23].
It is worth noting that the website ¡®Understand=
ing
Uncertainty¡¯ [24] of the team associated with David Spiegelhalter (FRS,
Professor of Public Understanding of Risk), independently also criticises w=
ork
[1] on the basis of RTM and potential publication bias.
It should be remembered that basic statistical
theory develops from the assumption that observations of a random variable =
are
independent and identically distributed (IID). That is the observations on a
given variable come from the same probability distribution and that each
¡®realisation¡¯ of it is unaffected by any other ¡®draw¡¯. However, in rega=
rd to
that last assumption of independence, in the real world what happens before=
may
influence what happens later; that is the observations may be clustered [8].
For a positive association this will increase the variance of an estimate (=
and
is therefore said to be over-dispersed). This is essentially because the
non-independence reduces the effective sample size. The issue of
over-dispersion is likely to be influential in studies of crime, as crime is
clustered within criminals due to repeat offending, and not important in the
occurrence of road collisions as these are more likely to occur in an
independent fashion. As stated, criminal behaviour is not independent but
clustered, causing higher variability than independence would predict. This=
is
because any criminal may have prolific activity. Therefore, such a person
starting and stopping criminal activity will give marked fluctuations in the
number of crimes committed. The clustering relates to the fact that the cri=
mes
are clustered within each criminal. (The count of crimes in a given area and
period will be the sum of all such activity). Contrast this with the situat=
ion
of repeatedly counting, in a fixed time period, the products of radioactive
decay of a piece of long-lived radioactive material. Each radioactive nucle=
us
neither knows nor cares about decaying, it just has the same probability of
decaying in each time period. This leads to the counts forming a Poisson
distribution. This has defined properties, all characterised simply by its
mean.
A further issue is that if the desire is to comb=
ine
the results of different studies, as in a meta-analysis, it is necessary to
ascribe study weights, which determine the magnitude of the contribution th=
at a
study gives to the pooled result. These weights are inversely proportional =
to
the variance, which measures the uncertainty around each study¡¯s estimate =
of
effect, and the over-dispersion may well be different, and indeed not well-=
known,
in different studies. This will give rise to additional uncertainty in the
pooled estimate of effect.
6. Post-Implementation
Surveillance.
There is a need to evaluate what actually has
happened after a policy has been implemented. In clinical research this
practise is given the name ¡®Phase IV trial¡¯ in which, after a treatment h=
as
been approved and it is in routine use, its long-term effects are studied o=
ver
a lengthy period. By this means rare, important side effects can be picked =
up,
as well as seeing how effective the treatment is in routine use.
In my view there ought to be a requirement to ch=
eck
results as a matter of routine for all public policy implementations. Note =
the
¡®roll out¡¯ of the policy should be designed so as to make its evaluation,=
after
implementation, straightforward, with the criteria set in advance. The proc=
ess
should be supervised by independent statisticians and scientists. Again
unintended effects of the policy might be picked up. As with all science,
transparency is important and so open access to data must be required here =
too.
Such checking ought always to be done after large sums of money have been
spent, as for example on implementing new road lighting. It is important to
know whether the money spent has produced the effects in terms of public sa=
fety
that it was supposed to. Perhaps the large sums of public money spent on ro=
ad
lighting could have been differently and better used in the pursuit of
enhancing public safety, without the environmental impact of more artificial
light. High quality checking can be done at a small fraction of the cost of=
the
policy implementation.
The LANTERNS project for investigating the effec=
t of
lighting reduction was funded, possibly due to a strong belief that road
lighting is essential to public safety. Therefore, if the belief is true a
reduction in lighting would be expected to have a negative effect on safety;
yet no such effect was found.
In contrast to LANTERNS, as yet no project has b=
een
set up specifically to evaluate the wide implementation of new road lightin=
g in
the UK to see whether it has fulfilled the expectations promised. Such an
evaluation might consider the issue of ¡®value for money¡¯ using the eviden=
ce it
obtained.
7. Science and Society
There is a big difference between true science a=
nd
public relations:
•
•
It is concerning that academic research seems to=
be
getting more public relations minded.
We need to avoid what Nobel laureate physicist
Richard Feynman called ¡®Cargo Cult Science¡¯ [27] by for example trying ¡°=
to give
all of the information to help others to judge the value of your contributi=
on;
not just the information that leads to judgment in one particular direction=
or
another¡±.
A cultural shift is needed towards thinking what
might reasonably be wrong with research as opposed to blindly accepting the
frenzied generation of more research findings of dubious correctness. We ne=
ed
to value the quality of the scientific basis of a study, rather than whethe=
r it
gives an impressive result which may be erroneous.
There is concern that the current academic system
encourages the thriving of bad science at the expense of the good by a proc=
ess
of natural selection [28]. There are some suggestions of how to improve mat=
ters
[29]. Concerns about dubious claims are not that new [30].
Francis Bacon more than 400 years ago wrote [31]=
of
the practice that those making claims present them in a way that might best
believed, and those receiving them are not sufficiently diligent in their
scrutiny.
¡°For as knowledges are now delivered, there is a =
kind
of contract of error between the deliverer and the receiver. For he that
delivereth knowledge desireth to deliver it in such form as may be best
believed, and not as may be best examined; and he that receiveth knowledge
desireth rather present satisfaction than expectant inquiry; and so rather =
not
to doubt, than not to err: glory making the author not to lay open his
weakness, and sloth making the disciple not to know his strength.¡±.
8. Conclusion
Much has become clear about the ways that science
can produce incorrect answers. There are some safeguards against this probl=
em
in place in some areas of study. However, the problems causing erroneous
results need to be more generally known. Measures to combat these faults ne=
ed
to be extended to other areas of study, including research on lighting, as
outlined above. Doing this should help maintain a good reputation for scien=
ce
and scientists.
The ultimate safeguard against policies that do =
not
do what they are supposed to is that everyone understands and becomes more
aware of the issues and is critical of any poor work done; especially that =
done
in the public¡¯s name and expense.
References
[1]
Beyer, F. R., & Ker, K. (2009). Street lighting for preventing road tra=
ffic
injuries. The Cochrane Library.
[2]
Steinbach, R., Perkins, C., Tompson, L., Johnson, S., Armstrong, B., Green,=
J.,
Edwards, P. et al. (2015). The effect of reduced street lighting on road
casualties and crime in England and Wales: controlled interrupted time seri=
es
analysis. Journal of epidemiology and community health, jech-2015.
[3]
Horton, R. (2015). Offline: What is medicine¡¯s 5 sigma. The Lancet, 385(99=
76),
1380. Available from:
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(15)60696-1/f=
ulltext?rss%3Dyes.
[4]
Academy of Medical Sciences. Reproducibility and reliability of biomedical
research. [Internet]. Acmedsci.ac.uk. 2016 (cited 8 December 2016). Availab=
le
from:
http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliab=
ility-of-biomedical-research/.
[5] Saving Science from the Scientists - - BBC Radio 4 (cited 8 December 2016). Available from: http://www.bbc.co.uk/programmes/b0742nzq/episodes/player. <= o:p>
[6]
Science Betrayed - - BBC Radio 4 (cited 8 December 2016). Available from:
http://www.bbc.co.uk/ programmes/b00znb98/episodes/player.
[7]
The Economist. Trouble at the lab (cited 8 December 2016); Available from:
http://www.economist.com/
news/briefing/21588057-scientists-think-science-self-correcting-alarming-de=
gree-it-not-trouble.
[8]
Marchant, P. R. (2004). A demonstration that the claim that brighter lighti=
ng
reduces crime is unfounded. British Journal of Criminology, 44(3), 441-447.=
[9] Marchant, P. (2005). Evaluating area‐wide crime‐reduction measures. Significance, 2(2), 62-65.<= o:p>
[10]
Marchant, P. (2005). Shining a light on evidence-based policy: street light=
ing
and crime.
[11]
Marchant, P. (2005). What works? A critical note on the evaluation of crime
reduction initiatives. Crime Prevention & Community Safety, 7(2), 7-13.=
[12]
Marchant, P. (2006). Investigating whether a crime reduction measure works.
Radical Statistics, 91, 41. Available from:
http://www.radstats.org.uk/no091/Marchant91.pdf.
[13]
Marchant, P. (2010). What is the contribution of street lighting to keeping=
us
safe? An investigation into a policy. Radical Statistics, 102, 32-42. Avail=
able
from: http://www.radstats.org.uk/no102/Marchant102.pdf.
[14]
Marchant, P. R. (2010). Have new street lighting schemes reduced crime in
London. Radical Statistics, 104, 32-42. Available from:
http://www.radstats.org.uk/no104/Marchant2_104.pdf.
[15]
Private Eye. (2005). Conflicts of Interest: Let there be light.1142.28.
[16]
Higgins, J. P., & Green, S. (Eds.). (2011). Cochrane handbook for
systematic reviews of interventions (Vol. 4). John Wiley & Sons. Availa=
ble
from: http://handbook.cochrane.org.
[17]
Wilkins, H. (2006). United Nations Secretary-General's High Level Panel on
System Wide Coherence and the environment: incremental change or UN reform?.
International journal of global environmental issues, 6(4), 285-300. Availa=
ble
from: http://www.politico.eu/wp-content/uploads/2016/09/HLP-Report-FINAL-Se=
pt-2016.pdf.
[18]
ACCOUNTS, P. (2013). Access to clinical trial information and the stockpili=
ng
of Tamiflu. Available from:
http://www.publications.parliament.uk/pa/cm201314/cmselect/cmpubacc/295/295=
02.htm.
[19]
AllTrials (cited 8 December 2016). Available from: http://www.alltrials.net=
./
[20]
Perkins, C., Steinbach, R., Tompson, L., Green, J., Johnson, S., Grundy, C.,
Edwards, P. et al. (2015). What is the effect of reduced street lighting on
crime and road traffic injuries at night? A mixed-methods study. Available
from:
https://www.journalslibrary.nihr.ac.uk/programmes/phr/11300402/#/.
[21]
Perkins, C., Steinbach, R., Tompson, L., Green, J., Johnson, S., Grundy, C.,
Edwards, P. et al. (2015). What is the effect of reduced street lighting on
crime and road traffic injuries at night? A mixed-methods study. Available
from: https://www.journalslibrary.nihr.ac.uk/phr/phr03110#/full-report.
[22]
Marchant, P.R. (2008).Regression towards the Mean. In: Darity WA (ed.)
International Encyclopaedia of Social Sciences. 2nd Edition Thomson Gale.
141-143.
[23]
Baxter, P.D., Marchant, P.R. (2010). The cross product ratio in the bivaria=
te
log normal distribution and gamma distribution, with an application to
non-randomised trials, Journal of Applied Statistics, 37(4), 529-536.
[24]
Understanding Uncertainty. Does street lighting really reduce fatal road
crashes by 2/3? (cited 8 December 2016).Available from:
http://understandinguncertainty.org/node/231.
[25]
Enhancing the QUAlity and Transparency Of health Research (cited 8 December
2016). Available from: www.equator-network.org.
[26]
Moher, D., Altman, D.G. (2015) Four Proposals to Help Improve the Medical
Research Literature. PLoS Medicine,12(9), e1001864. Available from:
http://journals.plos.org/plosmedicine/article?id=3D10.1371/journal. pmed.10=
01864.
[27]
Feynman, R. Cargo Cult Science: Some remarks on science, pseudoscience, and
learning how to not fool yourself. Caltech¡¯s 1974 commencement address. (c=
ited
8 December 2016). Available from: http://calteches.
library.caltech.edu/51/2/CargoCult.htm.&nb=
sp;
[28]
Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad
science. Royal Society Open Science, 3(9), 160384.
[29]
Ioannidis, J. P. (2014). How to make more published research true. PLoS Med,
11(10), e1001747.
[30]
Altman, D. G. (1994). The scandal of poor medical research. BMJ: British
Medical Journal, 308(6924), 283.
[31]
Bacon, F. (1885). The advancement of learning. Clarendon Press. Available
from:
www.gutenberg.org/files/5500/5500-h/5500-h.htm.
P. Marchant /International Journal of Sustainable
Lighting 20 (2017) 69-74