# retroharmonize
[](https://www.repostatus.org/#active)
[](https://cran.r-project.org/package=retroharmonize)
[](https://cran.r-project.org/package=retroharmonize)
[](https://cran.r-project.org/package=retroharmonize)
[](https://app.codecov.io/gh/dataobservatory-eu/retroharmonize)
[](https://github.com/dataobservatory-eu/retroharmonize/actions)
The goal of `retroharmonize` is to facilitate retrospective (ex-post)
harmonization of survey data in a reproducible manner. The package
provides tools for organizing the metadata, standardizing the coding of
variables, variable names and value labels, including missing values,
and for documenting all transformations, with the help of comprehensive
S3 classes.
Currently being generalized from problems solved in the not yet released
[eurobarometer](https://github.com/antaldaniel/eurobarometer) package
([doi](https://doi.org/10.5281/zenodo.3825700).)
## Installation
The package is available on [CRAN](https://CRAN.R-project.org):
``` r
install.packages("retroharmonize")
```
The development version can be installed from
[GitHub](https://github.com/) with:
``` r
# install.packages("devtools")
devtools::install_github("rOpenGov/retroharmonize")
```
You can download the manual in PDF for the [0.2.4
release](https://retroharmonize.dataobservatory.eu/retroharmonize_0.2.4.pdf)
but it is significantly different from 0.2.5.
## Survey harmonization
Surveys, i.e., systematic primary observation and data collections are
important data sources of both social and natural sciences. They are in
most cases the primary data sources of scientific research. Drawing
information from several surveys, conducted in different locations or in
different time can greatly enhance the inferential capacity of the
surveys, but it requires significant data processing and statistical
processing work. Our R software package offers a practical and
comprehensive solution to harmonizing the datasets and their codebooks.
Statistical matching is a related concepts that can take a harmonized
dataset further, for example, with creating new, statistically better,
unified weights. For these problems,
[StatMatch](https://CRAN.R-project.org/package=StatMatch) is a mature
solution in R.
Survey data harmonization refers to procedures that improve the data
comparability or the inferential capacity of multiple surveys. The
*retroharmonize* package support various data processing, documentation,
file/type conversion aspects of various retrospective survey
harmonization workflows (i.e. harmonization tasks related to surveys
that already have already been conducted, recorded into a coded file.)
From a technical perspective, the aim of the survey harmonization is to
create a single, tidy, joined harmonized dataset in the form of a data
frame that contains a row identifier, which is truly unique across all
observations, and which also contains the concatenated and harmonized
variables. We do this in a way that provides an unambiguous mapping of
numerical coded and labelled data, including special and missing data.
This way we avoid coercion that may lead to logical errors due to
syntactically correct, but logically inconsistent variable labelling in
across differently coded source files. Taking the harmonization to the
level of type harmonization to *numeric* and *factor* classes allows the
use of R’s powerful statistical packages that require numeric or factor
type input, and a wide range of survey output harmonization (harmonized
statistics and indicators.
For an extended overview of these problems with illustrations please
refer to the vignette [Survey
Harmonization](https://retroharmonize.dataobservatory.eu/articles/survey_harmonization.html).
### 1. Importing
Survey data, i.e., data derived from questionnaires or systematic data
collection, such as inspecting objects in nature, recording prices at
shops are usually stored databases, and converted to complex files
retaining at least coding, labelling metadata together with the data.
This must be imported to R so that the appropriate harmonization tasks
can be carried out with the appropriate R types.
### 2. Harmonization of concepts
After importing data with some descriptive metadata such as numerical
coding and labelling, we need to create a map of the information that is
in our R session to prepare a harmonization plan. We must find
information related to sufficiently similar concepts that can be
harmonized to be successfully joined into a single variable, and
eventually a table of similar variables must be joined.
We create a map of the measured concepts that needs to be harmonized,
for example, a binary sex variable with missing cases and a four-level
categorical variable on gender identification that has *other* and
*declined* options. See the vignette [Working With Survey
Metadata](https://retroharmonize.dataobservatory.eu/articles/survey_harmonization.html)
how mapping the metadata of the surveys can help getting started with
this first step.
We use a [crosswalk
table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.html)
or a *crosswalk scheme* for all the variable name, value label and type
conversion tasks that we plan to do.
### 3. Harmonization of variable names
Make sure that `survey_1$sex` and `survey_2$gender` can be concatenated
to a gender vector or `survey_joined$gender`. See more in the [Working
With A Crosswalk
Table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.html).
### 4. Harmonization of variable numerical codes and labels
For example, *Female=0* in `survey_1$sex` and *female=2* in
`survey_2$gender` becomes consistently female=0. Missing and *declined*
values are consistently handled.
### 5. Consistent types
To use R’s statistical functions with the concatenated version of
`survey_1$sex` and `survey_2$gender` they must have the same R type. In
the vast majority of the cases either *numeric* or *factor*, and in data
visualization applications sometimes *character*. See more in the
[Harmonize Value
Labels](https://retroharmonize.dataobservatory.eu/articles/harmonize_labels.html)
vignette.
### 6. Reproducibility & Documentation
To review statistical results and model results derived from the
concatenated variable (or the joined data frame), they must remain
comparable with `survey_1$sex` and `survey_2$gender`. It is also
necessary to have a new, unique row ID for each observation. If you want
to make your work available outside R, in a different software, the
joined, longitudional data frame must be exported in a consistent
manner.
## Use Cases
We also provide three extensive case studies illustrating how the
`retroharmonize` package can be used for ex-post harmonization of data
from cross-national surveys:
- [Afrobarometer](https://retroharmonize.dataobservatory.eu/articles/afrobarometer.html)
- [Arab
Barometer](https://retroharmonize.dataobservatory.eu/articles/arabbarometer.html)
- [Eurobarometer](https://retroharmonize.dataobservatory.eu/articles/eurobarometer.html)
The creators of `retroharmonize` are not affiliated with either
Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that
designs, produces or archives their surveys.
We create a large, harmonized dataset for extensive testing of our
packages capabilities. The replication data of this special use case can
be found on
- [Harmonized Cultural Access & Participation
Dataset](https://retroharmonize.dataobservatory.eu/articles/cap.html)
You can find this harmonized dataset on Zenodo in the [Digital Music
Observatory](https://zenodo.org/communities/music_observatory/) and the
[Cultural Creative Sectors Industries Data
Observatory](https://zenodo.org/communities/ccsi/) repositories.
[](https://doi.org/10.5281/zenodo.5781672)
We are building experimental APIs data in the form of automated
observatories, which are running retroharmonize regularly and improving
known statistical data sources. See also the [Green Deal Data
Observatory](https://greendeal.dataobservatory.eu/) and the [Economy
Data Observatory](https://economy.dataobservatory.eu/).
## Working with SPSS files
Survey data is often available in SPSS’s custom labelled format.
Unfortunately, joining data with different labelling is not possible.
When you do not need to preserve the history of complex harmonization
problems, codebook, etc, then you do not necessary need to look under
the hoods of our S3 classes. The new
[`labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
class is an inherited extension of [haven’s labelled_spss
class](https://haven.tidyverse.org/reference/labelled_spss.html). It not
only preserves variable and value labels and the user-defined missing
range, but also gives an identifier, for example, the filename or the
wave number, to the vector. Additionally, it enables the preservation—
as metadata attributes—the original variable names, labels, and value
codes and labels, from the source data. This way, the harmonized data
also contain the pre-harmonization record. The vignette [Working With
The labelled_spss_survey
Class](https://retroharmonize.dataobservatory.eu/articles/labelled_spss_survey.html)
provides more information about the
[`labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
class.
In [Harmonize Value
Labels](https://retroharmonize.dataobservatory.eu/articles/harmonize_labels.html)
we discuss the characteristics of the
[`labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
class and demonstrates the problems that using this class solves.
## Citations and related work
### Citing the data sources
Our package has been tested on three harmonized survey’s microdata.
Because [retroharmonize](https://retroharmonize.dataobservatory.eu/) is
not affiliated with any of these data sources, to replicate our
tutorials or work with the data, you have download the data files from
these sources, and you have to cite those sources in your work.
**Afrobarometer** data: Cite
[Afrobarometer](https://www.afrobarometer.org/data/) **Arab Barometer**
data: cite [Arab
Barometer](https://www.arabbarometer.org/survey-data/data-downloads/).
**Eurobarometer** data: The
[Eurobarometer](https://commission.europa.eu/resources/statistics/public-opinion-survey_en)
data
[Eurobarometer](https://commission.europa.eu/resources/statistics/public-opinion-survey_en)
raw data and related documentation (questionnaires, codebooks, etc.) are
made available by *GESIS*, *ICPSR* and through the *Social Science Data
Archive* networks. You should cite your source, in our examples, we rely
on the [GESIS](https://search.gesis.org/) data files.
### Citing the retroharmonize R package
For main developer and contributors, see the
[package](https://retroharmonize.dataobservatory.eu/) homepage.
This work can be freely used, modified and distributed under the GPL-3
license:
``` r
citation("retroharmonize")
#> To cite package 'retroharmonize' in publications use:
#>
#> Antal D (2026). _retroharmonize: Ex Post Survey Data Harmonization_.
#> R package version 0.2.6,
#> .
#>
#> A BibTeX entry for LaTeX users is
#>
#> @Manual{,
#> title = {retroharmonize: Ex Post Survey Data Harmonization},
#> author = {Daniel Antal},
#> year = {2026},
#> note = {R package version 0.2.6},
#> url = {https://retroharmonize.dataobservatory.eu/},
#> }
```
### Contact
For contact information, see the
[package](https://retroharmonize.dataobservatory.eu/) homepage.
### Code of Conduct
Please note that the `retroharmonize` project is released with a
[Contributor Code of
Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/).
By contributing to this project, you agree to abide by its terms.
# Package index
## Importing
Survey data, i.e., data derived from questionnaires or systematic data
collection, such as inspecting objects in nature, recording prices at
shops are usually stored databases, and converted to complex files
retaining at least coding, labelling metadata together with the data.
This must be imported to R so that the appropriate harmonization tasks
can be carried out with the appropriate R types.
- [`read_surveys()`](https://retroharmonize.dataobservatory.eu/reference/read_surveys.md)
[`read_survey()`](https://retroharmonize.dataobservatory.eu/reference/read_surveys.md)
: Read survey files into memory or save as \`.rds\`
- [`read_rds()`](https://retroharmonize.dataobservatory.eu/reference/read_rds.md)
: Read a survey from an \`.rds\` file
- [`read_spss()`](https://retroharmonize.dataobservatory.eu/reference/read_spss.md)
: Read SPSS survey files
- [`read_dta()`](https://retroharmonize.dataobservatory.eu/reference/read_dta.md)
: Read a Stata \`.dta\` survey file
- [`read_csv()`](https://retroharmonize.dataobservatory.eu/reference/read_csv.md)
: Read a survey dataset from a CSV file
- [`pull_survey()`](https://retroharmonize.dataobservatory.eu/reference/pull_survey.md)
: Retrieve a survey from a survey list
## Harmonizing concepts with metadata
After importing data with some **descriptive metadata** such as
**numerical coding** and **labelling**, we need to create a map of the
information that is in our R session to prepare a harmonization plan. We
must find information related to sufficiently similar concepts that can
be harmonized to be successfully joined into a single variable, and
eventually a table of similar variables must be joined.
- [`metadata_create()`](https://retroharmonize.dataobservatory.eu/reference/metadata_create.md)
[`metadata_waves_create()`](https://retroharmonize.dataobservatory.eu/reference/metadata_create.md)
: Create metadata tables from survey datasets
- [`metadata_survey_create()`](https://retroharmonize.dataobservatory.eu/reference/metadata_survey_create.md)
: Create variable-level metadata from a survey dataset
- [`retroharmonize`](https://retroharmonize.dataobservatory.eu/reference/retroharmonize.md)
: retroharmonize: Retrospective harmonization of survey data files
## Codebooks
The new functions will follow the DDI and SDMX terminology. See vignette
[Harmonizing Concepts, Questions, and
Variables](https://retroharmonize.dataobservatory.eu/articles/concept.html)
- [`create_codebook()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
[`codebook_waves_create()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
[`codebook_surveys_create()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
: Create a survey codebook
## Harmonize variable names
Before joining variables containing responses about the same concept,
make sure that they have **identical names** in the re-processed
surveys. See the vignette [Working with a Crosswalk
Table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.html)
for examples and further clarification.
- [`harmonize_var_names()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_var_names.md)
: Harmonize the variable names of surveys
- [`label_normalize()`](https://retroharmonize.dataobservatory.eu/reference/label_normalize.md)
[`var_label_normalize()`](https://retroharmonize.dataobservatory.eu/reference/label_normalize.md)
[`val_label_normalize()`](https://retroharmonize.dataobservatory.eu/reference/label_normalize.md)
: Normalize value and variable labels
- [`harmonize_survey_variables()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_survey_variables.md)
: Read a survey from a CSV file
## Harmonize numerical codes and labels
To merge variables from different surveys into a single variable, you
must make sure that the numerical codes and labels, for example *0=‘no’*
and *1=‘yes’* are processed identically. See the vignette [Harmonize
Value
Labels](https://retroharmonize.dataobservatory.eu/articles/harmonize_labels.html)
for examples and further clarification.
- [`collect_val_labels()`](https://retroharmonize.dataobservatory.eu/reference/collect_val_labels.md)
[`collect_na_labels()`](https://retroharmonize.dataobservatory.eu/reference/collect_val_labels.md)
: Collect labels from metadata file
- [`harmonize_values()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_values.md)
: Harmonize the values and labels of labelled vectors
- [`harmonize_survey_values()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_survey_values.md)
[`harmonize_waves()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_survey_values.md)
: Harmonize values in surveys
- [`merge_surveys()`](https://retroharmonize.dataobservatory.eu/reference/merge_surveys.md)
: Merge and harmonize surveys
- [`merge_waves()`](https://retroharmonize.dataobservatory.eu/reference/merge_waves.md)
: Deprecated wrapper for \`merge_surveys()\`
## Harmonize missing and special cases
Some variable codes have a special meaning, such as a various labels of
**missing values** which need to be converted differently to numeric,
factor or character representation. See the vignette [Harmonize Value
Labels](https://retroharmonize.dataobservatory.eu/articles/harmonize_labels.html)
for examples and further clarification.
- [`collect_val_labels()`](https://retroharmonize.dataobservatory.eu/reference/collect_val_labels.md)
[`collect_na_labels()`](https://retroharmonize.dataobservatory.eu/reference/collect_val_labels.md)
: Collect labels from metadata file
- [`na_range_to_values()`](https://retroharmonize.dataobservatory.eu/reference/na_range_to_values.md)
: Harmonize SPSS-style missing value ranges
- [`harmonize_na_values()`](https://retroharmonize.dataobservatory.eu/reference/harmonize_na_values.md)
: Harmonize na_values in haven_labelled_spss
## Crosswalk
Laying out the harmonization **crosswalk scheme** (unifying variable
names, codes, labels.) See the vignette [Working with a Crosswalk
Table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.html)
for examples and further clarification.
- [`is.crosswalk_table()`](https://retroharmonize.dataobservatory.eu/reference/crosswalk_table_create.md)
[`crosswalk_table_create()`](https://retroharmonize.dataobservatory.eu/reference/crosswalk_table_create.md)
: Validate a crosswalk table
- [`crosswalk_surveys()`](https://retroharmonize.dataobservatory.eu/reference/crosswalk_surveys.md)
[`crosswalk()`](https://retroharmonize.dataobservatory.eu/reference/crosswalk_surveys.md)
: Crosswalk and harmonize surveys
## Subsetting
Remove variables that **cannot be harmonized** in your workflow either
in memory (faster for smaller tasks) or sequentially from files. See the
vignette [Working with a Crosswalk
Table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.html)
for examples and further clarification.
- [`subset_surveys()`](https://retroharmonize.dataobservatory.eu/reference/subset_surveys.md)
[`subset_waves()`](https://retroharmonize.dataobservatory.eu/reference/subset_surveys.md)
[`subset_save_surveys()`](https://retroharmonize.dataobservatory.eu/reference/subset_surveys.md)
: Subset and optionally harmonize surveys
## Documentation functions
These functionality requires a thorough review.
- [`document_survey_item()`](https://retroharmonize.dataobservatory.eu/reference/document_survey_item.md)
: Document survey item provenance
- [`document_surveys()`](https://retroharmonize.dataobservatory.eu/reference/document_surveys.md)
[`document_waves()`](https://retroharmonize.dataobservatory.eu/reference/document_surveys.md)
: Document survey lists
- [`create_codebook()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
[`codebook_waves_create()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
[`codebook_surveys_create()`](https://retroharmonize.dataobservatory.eu/reference/create_codebook.md)
: Create a survey codebook
## Type conversion
Consistently treat labels, missing value ranges, missing value labels
imported from SPSS, STATA or other source to use R language statistical
functions, which mainly work with the base class of **numeric** or
**factor**. For data visualization, the base class **character** may be
preferred. See vignette [The labelled_spss_survey
class](https://retroharmonize.dataobservatory.eu/articles/labelled_spss_survey.html)
for further information.
- [`survey()`](https://retroharmonize.dataobservatory.eu/reference/survey.md)
[`is.survey()`](https://retroharmonize.dataobservatory.eu/reference/survey.md)
[`summary(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/survey.md)
: Create a survey data frame
- [`is.survey_df()`](https://retroharmonize.dataobservatory.eu/reference/survey_df.md)
[`survey_df()`](https://retroharmonize.dataobservatory.eu/reference/survey_df.md)
[`print(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/survey_df.md)
: Create a survey object
- [`labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`` `[`( ``*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`print(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`summary(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`is.na(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`levels(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`` `names<-`( ``*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`format(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`is.labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`median(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`quantile(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`weighted.mean(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`mean(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
[`sum(`*``*`)`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey.md)
: Labelled SPSS-style vectors with survey provenance
- [`as_labelled_spss_survey()`](https://retroharmonize.dataobservatory.eu/reference/as_labelled_spss_survey.md)
: Labelled to labelled_spss_survey
- [`concatenate()`](https://retroharmonize.dataobservatory.eu/reference/concatenate.md)
: Concatenate haven_labelled_spss vectors
- [`as_numeric()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey_coercion.md)
[`as_character()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey_coercion.md)
[`as_factor()`](https://retroharmonize.dataobservatory.eu/reference/labelled_spss_survey_coercion.md)
: Coercion methods for labelled survey vectors
# Articles
### Conceptual
We try to bulid a correspondance with the DDI and SDMX standards, the
exiting literature and our own practice.
- [Harmonizing Concepts, Questions, and
Variables](https://retroharmonize.dataobservatory.eu/articles/concept.md):
- [Value Labels and
Codelists](https://retroharmonize.dataobservatory.eu/articles/codelist.md):
- [Working with a Crosswalk
Table](https://retroharmonize.dataobservatory.eu/articles/crosswalk.md):
- [Harmonize Value
Labels](https://retroharmonize.dataobservatory.eu/articles/harmonize_labels.md):
### Tutorials
An overview of the harmonization workflow and support for **mapping
concepts**, creating a **crosswalk scheme** of the harmonization,
carrying out variable name, numerical code, labeling and type setting
harmonization.
- [Survey
Harmonization](https://retroharmonize.dataobservatory.eu/articles/survey_harmonization.md):
- [Working With Survey
Metadata](https://retroharmonize.dataobservatory.eu/articles/metadata.md):
- [documentation](https://retroharmonize.dataobservatory.eu/articles/documentation.md):
- [Getting
Started](https://retroharmonize.dataobservatory.eu/articles/retroharmonize.md):
### Use cases
Real-life examples with thee major harmonized survey program’s data.
- [Case Study: Working With Afrobarometer
surveys](https://retroharmonize.dataobservatory.eu/articles/afrobarometer.md):
### Under the hood
The s3 class inheritted from *labelled* and *haven*
- [The labelled_spss_survey
class](https://retroharmonize.dataobservatory.eu/articles/labelled_spss_survey.md):