Skip to main content
University of York Library
Subject Guides

Skills Guides

Data sources

Feedback
X

Data and stats - what are they and what can I do with them?

We often use these two words interchangeably, and the word data is often used in a broad sense to refer to a collection of facts, numbers, and statistics collected together. In academic research the two are different and the difference is important in order to understand exactly what you are looking at and, crucially, what you can do with it.

Data

  • Raw numbers collected as part of a study and stored.

  • Usually found in the form of a digital dataset -  a collection of related sets of information/data kept as machine-readable files, that can be filtered and searched according to your own criteria, and can be analyzed using software such as Excel and SPSS.

When is data useful?

Some organisations and researchers collect and store enormous quantities of data that can be used by people as a basis for reasoning, discussion or calculation. People can interrogate and analyse this data in order to explain things.

Data series are often kept over long periods of time and updated regularly with the most recent figures. This means they can be repeatedly interrogated, each time potentially showing new developments. You can start to compare a variable over time and be very up-to-date in what you find.

Data is particularly useful when you want to examine what is happening and come to your own conclusions, i.e. when you are asking questions like 'Why?' Or 'How?'


Statistics

Statistics are the result of some human analysis of the raw collected data. Data has been interrogated and processed in some way and decisions have been made on how to present that data to show a particular view of what is going on. 

You will usually see statistics in tables, charts, or graphs, and also as numbers and percentages reported in articles. 

Once a statistic is published it is static and only ever refers to that point in time.

When are stats useful?

  • When you need a quick figure to evidence a point, and you think that figure may already have been pulled out for you by someone else publishing on the topic or by an organisation who pulls together summary statistics (e.g. GDP for a country in a particular year, % of people who enrolled in higher education in a particular country; number of crimes reported in a particular place and time) and you don't want to interrogate the data for yourself in your own way.

  • When you want to answer questions like 'how many?' or 'how much?'

"Lies, damned lies, and statistics"

Pinocchio marionettes for sale

Statistics can seem persuasive, but beware. They can often be used to make a weak argument seem stronger. Think critically about statistics that are presented to you, and decide whether you think they are strong enough as evidence to prove an argument. Think about what that stat really tells you and what it leaves out.

For more on how to read statistics with a critical eye, see the book:
"Damned lies and statistics: untangling numbers from the media, politicians, and activists", by Joel Best.

Data sources

We subscribe to a number of statistical data sources, including international statistics, datasets, and market reports. Find out more via the links below:

Ready to search? Getting hold of data

There are many databases that specialise in making data available and searchable. Often these are produced by the people who gather the data, and the date included in each will depend on what is of interest to the organisation/s that have gathered it.

Some of the larger and more comprehensive databases for social and economic data are good places to start:

You may find useful statistics reported in journal articles. Someone else may have already done research in the area you're interested in. Scholars usually publish their findings in journal articles, including some of the data. You will also find stats in newspapers and magazines. Do be critical of your sources and follow them up though to be sure they are reputable and authoritative sources though.

The Library subscribes to many databases for searching for journal articles and other secondary sources. See your own Library Subject Guide for which databases to try when searching for journal articles.

Searching the web can be a minefield. Here sre some tips for searching on the web for statistics:

  • Add in words like data or statistics to your search terms

  • You can search particular sites or domains using advanced search functionality. For example, the site command in Google. A search for site:gov entered in with your keywords will only search sites with gov in the URL. See individual Help sections on Search Engines for more information.

  • YorSearch and Library catalogues tell you about particular titles a Library has in stock, some of which may be statistical in nature. The word statistics will be in the subject terms field, so you can use this word in a subject terms search in the advanced search. 

 

Forthcoming training sessions

Forthcoming sessions on :

Show details & booking for these sessions

There's more training events at:

Gathering data

We take a look at what data is and where we might get it, from surveys to pre-existing data sources.

Digging for data

Do you need to find published data to explore and support arguments in your work? Are you struggling to find the web’s most authoritative data sources? Here we look at a range of resources including the Economic and Social Data Service, and UN and OECD statistics.

Questions to frame your search for data

You will probably be looking for data about certain people, activities or commodities.

If you're looking for data on people, you need to think about what 'social unit' you're interested in. You may be interested in individuals, couples, households or families. You may be after particular groups, like companies or political organisations. You may be interested in a nation. Particular groupings may also be important to your search — for example, race, nationality or gender. 

If may be that you're interested in 'things' rather than people: for example, commodities like cars. Or you may be interested in a particular activity, like voting.

Defining exactly what is in scope and what is out of scope before you start searching will save you time later.

  • Are you interested in a particular point in time? 
  • Do you want to look at changes in data over time? 
  • Do you need historical information? 
  • Or do you want the most current information?

Most current statistics may actually be a year or more old as there are time lags whilst the information is collected, processed, and released.

Some sources of data offer huge banks of data that are gathered over years and you can interrogate these to do over-time comparisons. 

Some sources just offer quick snapshots of a particular point in time.

  • Are you interested in a particular nation, state, county or city?

Geography is usually a factor. There is usually a place that is the focus of your enquiry.

 

In order to find the right source to search for your data you need to consider the following:

  • Who is likely to collect the data on this? What kind of organisation?
  • Who is likely to process the data and publish this either as data or stats?

Who publishes statistical information?

  • Government departments - they collect data to aid them with policy decisions. This can be ministerial and non-ministerial departments.

  • Not-for-profit organisations - there are organisations who collect and publish styatistics to support their own agendas or aims. E.g. the World Health Organisation or the World Monetary Fund.

  • Commercial firms - for example marketing companies.

  • Academics - researchers and institutions gather and publish data as part of research projects. You can often find journal articles about topics that contain statistics as evidence. But you may be able to get access to whole datasets.

You may need to be a detective and do some Internet searching to determine who you think the main people are with a vested interest in your topic area. Work out if they publish data on their own websites. 

If you are interested in particular countries it is a good idea to work out the government structure in order to work out who will likely publish stats on particular themes.

I can't find this stat! What can I do?

Flexibility and detective work are essential. 

  • What you are looking for may never have been collected or may not be published for people to see. This is true of some countries where the data is just not made available. It may be that if it is very current information it is not available yet, or if it is older it may no longer be available to view. Can you refine your topic of investigation, changing either the geographical area, the timeframe or the other variables in some way?
  • Have you used the specialist tools and drawn a blank? Can you identify other organisations who may collect that data and explore their websites? Can you explore journal articles and other secondary sources instead?
  • Are you using the tool you're using to search effectively? Information databases and web search engines have help sections and tutorials with search tips.

Data formats

Data can be organised in a number of standardised and interoperable text-based formats. Whether you're importing existing data, or exporting it for use in another tool, it's worth understanding the common formats in use.

Delimited (CSV and TSV)

An archival standard for spreadsheet-formatted data is to use delimited text: each cell is separated by a special character (usually a comma or a tab character), and each row by a different character (usually a line-break character).

The most common delimited formats are CSV (comma-separated values) and TSV (tab-separated values):

CSV

Forename,Surname,Year,College
David,Jones,2,Derwith
Farrokh,Bulsara,1,Alcricke
Catherine,Bush,2,Langbrugh

TSV

Forename    Surname    Year    College
David    Jones    2    Derwith
Farrokh    Bulsara    1    Alcricke
Catherine    Bush    2    Langbrugh
Tip

Spreadsheet files can be saved into these formats for use elsewhere, but only the superficial text values are saved, not any formatting or underlying formulae.

Cells containing commas (or tabs) are encoded in quotation marks. If your data contains complicated combinations of commas (or tabs) and quotation marks, you may have problems saving as csv (or tsv), though you could potentially save with a different delimiter!


XML

XML (eXtensible Markup Language) uses nested tags (and corresponding closing tags) to define data and relationships. This webpage is essentially built using the same principles.

Tip

XML can incorporate rules (Document Type Definitions) to determine the data type of any given field. But because you don’t have to, and because it's quite difficult to do, it doesn't always happen.

While delimited filetypes can organise data in two dimensions (rows and columns), XML can encode more elaborate relational data.

Tip

XML files can often be imported into Excel, but the relationships contained may be too complicated to display adequately in a spreadsheet.

XML

<PEOPLE>
<PERSON>
<FORENAME>David</FORENAME>
<SURNAME>Jones</SURNAME>
<YEAR>2</YEAR>
<COLLEGE>Derwith</COLLEGE>
</PERSON>
<PERSON>
<FORENAME>Farrokh</FORENAME>
<SURNAME>Bulsara</SURNAME>
<YEAR>1</YEAR>
<COLLEGE>Alcricke</COLLEGE>
</PERSON>
<PERSON>
<FORENAME>Catherine</FORENAME>
<SURNAME>Bush</SURNAME>
<YEAR>2</YEAR>
<COLLEGE>Langbrugh</COLLEGE>
</PERSON>
</PEOPLE>

JSON

Like XML, JSON (JavaScript Object Notation) uses a hierarchical structure, in this case arranged in name/value pairs. The brackets and quote-marks may make JSON a little harder to read than XML, but the lack of closing tags means that files are a lot shorter (something that's quite important when you've got a lot of data). It's also easier to parse for use with a coding language.

JSON

{"people":[
{ "forename":"David", "surname":"Jones", "year":2, "college":"Derwith" },
{ "forename":"Farrokh", "surname":"Bulsara", "year":1, "college":"Alcricke" },
{ "forename":"Catherine", "surname":"Bush", "year":2, "college":"Langbrugh" }
]}
Tip

The latest versions of Excel can open JSON files, but not yet in an especially useful way. You'd generally interrogate JSON with a coding language like JavaScript or Python.

Getting data from web tables and PDFs

Not all data you find online is in a friendly format. You may occasionally come across tables of useful statistics on webpages or in PDFs. Sometimes you can copy and paste them into something like a spreadsheet without any problems, but not always.


Web tables

If data on a webpage has been formatted as a table or a list, and copying and pasting isn't pulling information across as you'd like, you should be able to import the data into a spreadsheet using an import function. Even if the data has been formatted in a non-standard way, you may still be able to extract usable information using an import function like IMPORTXML in Google or WEBSERVICE in Excel, but you might have to dig a bit deeper into the HTML.


PDFs

So long as the data in the PDF is encoded as text (rather than as an image), it can be extracted into a spreadsheet format. On University computers you can use ABBYY FineReader to convert a PDF to Excel format. If you're on your own machine you could use Google Drive to convert the PDF to a Google Doc, and then copy and paste. There are also free tools like Tabula, though, as ever, you should think critically when using software from the internet.

If your data is just an image (a photograph or photocopy of some data, with no machine-readable element), you'll need to employ some optical character recognition (OCR). If you're on campus, the scanning options on the printers/photocopiers includes OCR. Alternatively, you could use Google Drive to convert a PDF to a Google Doc. Either way, the results may not be structured in a very useful way, and you may have to do a lot of repair. It may be easier to simply enter the data yourself.

Cleaning data

It's one thing finding some data, but you probably need to manipulate it in some way before you can interrogate it...

Understanding the data you have

We might think of ‘data’ as values stored without context. Through processing that data we can seek to provide context and determine meaning. But even simple spreadsheet operations require us to have some understanding of what's in that dataset, and what constitutes ‘good’ data in the first place.

As an example, let’s ‘deconstruct’ some information:

“The appointment with Dr Watt is on Tuesday at 2:30pm at the Heslington Lane surgery.”

This information contains the following fields of data:

  • Who the appointment is with
  • The day (date) of the appointment
  • The time of the appointment
  • The location of the appointment

If you wanted to record appointments in a computer-based system you would need to use separate ‘fields’ for these — which in a spreadsheet might translate to separate columns.

When faced with an existing dataset, our first challenge might well be to reverse this process and rebuild our understanding of what information these fields convey. If you've got the data from a third-party source, look out for any explanatory notes that might help you with this.

Data types

Data processing systems struggle if you don’t stick to recognised data types, or if you add in values that don’t match others in the same context, For instance, in addition to text, spreadsheets observe the following special data types:

Data typeGoodUnrecognisable
Number5
1.6
-350
0.105
About 10
>5
10-15
25cm
Date/time01/01/2000
23-11-1963
15:30
17:16:20
01.01.2000
Mon or Fri
Next Tuesday
About 10:30
BooleanTrue
False
Maybe
?

For software to be able to analyse a number or a date, it needs a number or a date that it can parse — that it can understand and calculate with. If a value doesn't match the necessary rules to qualify as 'parsable', it will be treated as text. This may have an affect on how you're able to interrogate that data. If you represent a number or date in a way that does not allow the program to determine its type correctly, you will not be able to sort and filter correctly, you will not be able to add up, find averages, find the interval between two dates, etc... You might be able to understand that 20 + c.10 = c.30, but a computer can't make that leap. You're going to have to clean your data.

Cleaning

The success of any data processing will depend in large part on the quality of the source data you're working with. Data is often messy: columns might contain a mix of text and numerical data; some rows may have missing data; or perhaps you're trying to mash together two separate spreadsheets and the column names don’t quite match, or people have used a label in slightly different ways.

This is when you need to clean your data (a process also known as data munging or data wrangling). You need your data to be in a useful shape for your needs: if you're analysing or visualising data, what information (and types of data) does that analysis or visualisation require?

It’s all about ensuring that your data is validated and quantifiable. For instance, if you have a column of 'fuzzy' dates (e.g. c.1810 or 1990-1997), you might want to create a new column of 'parsed' dates — dates that are machine-readable (e.g. 1810, 1990). This might mean that you're losing some information and nuance from your data, and you'll need to keep that in mind in your analysis. But you'll at least have quantifiable data that you can analyse effectively.

For small, straightforward datasets, you can do data cleaning in a spreadsheet: ensure that numbers and dates are formatted as their appropriate data type, and use filters to help you standardise any recurring text. Excel even has a Query Editor tool that makes a lot of this work even easier.

The larger a dataset, the harder it is to work with it in a spreadsheet. Free tools like OpenRefine offer a relatively friendly way to clean up large amounts of data, while programming languages like R and Python have functions and libraries that can help with the tidying process.

Data structures

The way your data is laid out has an impact on how you can analyse it.

'Flat file' and relational data

Data is conventionally displayed as a two-dimensional table (rows and columns). Generally this will be laid out as a relationship between a case (a 'tuple') in each row, and its corresponding attributes (each with their own data type) in columns. Take this example of list structured data from a student fundraiser:

ABCDEFGHI
1Student ID ForeameSurnameYearCollegeBean bath10k runParachute jumpTandem joust
21001 DavidJones2Derwith60.0075.50 55.00
31002 FarrokhBulsara1Alcricke70.0085.0045.50
41003 CatherineBush2Langbrugh 65.5095.5035.00

Sometimes a single 'flat file' table of rows and columns is not enough. For instance:

You need to work with information about people and the research projects they are involved in. There will be several fields of data about the people, but also several about the projects.

It would be impossible to design one table that is suitable to hold all the data about people and projects, so in this case we create separate tables – one for people and one for projects – and find ways to express the connections between them.

In this example, one person can be involved in many projects, and one project can involve many people. This is a clear indication that the data is relational, and any attempt to work with it using a simple table will entail compromises.

This approach marks out the fundamental difference between a spreadsheet and a relational database.


Reshaping your data

Even the fundraising example in the table above may be better thought of as multiple tables: one table could index the students alongside their forenames, surnames, year, and college; a second table could list all the bean bathers (by Student ID) and the corresponding amount raised; a third could list the 10k runners, etc.

Depending on the analysis you need to do, it may be necessary to restructure your data. One common approach is to reorganise your data into what we might call a 'pivotable' format.

In our student fundraiser example, we have multiple columns all sharing the same attribute: amount raised. We might therefore look to move all these values into a single column:

ABCDEFG
1 Student IDForenameSurnameYearCollegeActivityAmount
21001 DavidJones2DerwithBean bath60.00
31001 DavidJones2Derwith10k run75.50
41001 DavidJones2DerwithTandem joust55.00
51002 FarrokhBulsara1AlcrickeBean bath70.00
61002 FarrokhBulsara1AlcrickeParachute jump45.00
71002 FarrokhBulsara1AlcrickeTandem joust85.00
81003 CatherineBush2Langbrugh10k run65.50
91003 CatherineBush2LangbrughParachute jump95.50
91003 CatherineBush2LangbrughTandem joust35.00

This table looks unusual when we're used to seeing one row per student. Now it's effectively one row per fundraising performance (we might even imagine a unique ID ascribed to each activity a student performs). But it means that all the fundraising amounts are now in the same column (G): we can get a total for that column very easily, and can even filter based on the activity, the student, or any other field. If we're using a spreadsheet, we can use this data in a pivot table, and if we're looking to make a visualisation, this is also the ideal format for a lot of visualisation tools.

Restructuring data is not always straightforward. But some of the data wrangling tools below may help you. We've also got some guidance on using spreadsheets to unpivot 'pivoted' data.

Maths skills

Forms & surveys