{
"cells": [
{
"cell_type": "markdown",
"id": "17f99679-6680-4b96-a978-af765635b0d8",
"metadata": {},
"source": [
"# Measurement Set Tutorial\n",
"\n",
"This tutorial notebook can be run on the Google Colaboratory environment via this [link](https://colab.research.google.com/github/casangi/xradio/blob/main/docs/source/measurement_set/tutorials/measurement_set_tutorial.ipynb).\n",
"\n",
"**XRADIO** is an open-source Python package that leverages [xarray](https://github.com/pydata/xarray) to provide an interface for radio astronomy data. It includes converters for legacy formats and defines versioned schemas for each dataset type. A schema checker is also included to verify whether a dataset conforms to the expected schema.\n",
"\n",
"Data is organized into the following structures:\n",
"\n",
"- [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html): A multi-dimensional, in-memory array container for labeled n-dimensional arrays.\n",
"- [xarray.DataArray](https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html): An n-dimensional array with labeled coordinates and dimensions.\n",
"- [xarray.DataTree](https://docs.xarray.dev/en/latest/generated/xarray.DataTree.html): A tree-like hierarchical collection of xarray objects (datasets and children trees).\n",
"\n",
"XRADIO extends the functionality of these xarray data structures by making use of **xarray accessors**.\n",
"\n",
"To use XRADIO effectively, it's important to understand the terminology and indexing methods used in xarray. The following documentation is recommended reading:\n",
"\n",
"- [xarray terminology](https://docs.xarray.dev/en/latest/user-guide/terminology.html)\n",
"- [xarray data structures](https://docs.xarray.dev/en/latest/user-guide/data-structures.html) (arrays, coordinates, datasets, and data trees)\n",
"- [xarray indexing and selection guide](https://docs.xarray.dev/en/latest/user-guide/indexing.html)\n",
"- [xarray accessors](https://docs.xarray.dev/en/latest/internals/extending-xarray.html)\n",
"\n",
"## Acronyms Used\n",
"\n",
"- `ps` = processing set \n",
"- `ms` = measurement set \n",
"- `cor` = correlated \n",
"- `xdt` = xarray DataTree \n",
"- `xds` = xarray Dataset \n",
"- `xda` = xarray DataArray \n",
"\n",
"Using these acronyms one can refer for example to the Processing Set DataTree (`ps_xdt`), the Measurement Set DataTree (`ms_xdt`) or the correlated dataset (`cor_xds`)."
]
},
{
"cell_type": "markdown",
"id": "75d18a3b-e0fc-452d-b65a-2d1bab66412d",
"metadata": {},
"source": [
"## Measurement Set v4 Schema Layout\n",
"\n",
"The Measurement Set v4 (MSv4) schema for radio telescope interferometric data (*Visibilities*) and single-dish data (*Spectrum*) is implemented in memory as an [`xarray.DataTree`](https://docs.xarray.dev/en/latest/generated/xarray.DataTree.html). \n",
"\n",
"The root node of this data tree (the measurement set `xarray.DataTree`, `ms_xdt`), is the `cor_xds` (correlated `xarray.Dataset`), which contains the Visibility or Spectrum data. It also includes data that shares the same dimensions as the Visibility/Spectrum data and serves a similar role to the *Main* table in Measurement Set v2 (MSv2).\n",
"\n",
"The child nodes of `ms_xdt` contain metadata such as antenna positions, field and source definitions, and other auxiliary information. Each `ms_xdt` is limited to a single observation, spectral window, polarization setup, observation mode, processor, and beam per antenna.\n",
"\n",
"Multiple `ms_xdt` trees can be grouped together as children of a processing set data tree (`ps_xdt`). This tree has an empty root node, with each child node representing an individual `ms_xdt`. The design of `ps_xdt` is inspired by the [Multi-MS format](https://casadocs.readthedocs.io/en/v6.2.0/notebooks/parallel-processing.html#The-Multi-MS) in MSv2.\n",
"\n",
"Figure 1 illustrates a simplified comparison between the Processing Set structure in MSv4 and the Multi-MS structure in MSv2. Blocks with the same color indicate similar information content. Blocks in between parenthesis are optional in their respective schemas.\n",
"\n",
"\n",
"
\n",
" \n",
" \n",
" Figure 1: Schema layouts of the Processing Set (collection of MSv4s) and the \n",
" Multi Measurement Set (MSv2).\n",
" \n",
" \n",
"
\n"
]
},
{
"cell_type": "markdown",
"id": "59444183",
"metadata": {},
"source": [
"### Access Pattern Cheat Sheet\n",
"\n",
"```python\n",
"import xarray as xr\n",
"import xradio # Enables accessors\n",
"from xradio.measurement_set import open_processing_set\n",
"\n",
"# Open a processing set data tree\n",
"ps_xdt = open_processing_set(\"my_ps.ps.zarr\") # or xr.open_datatree(\"...\")\n",
"\n",
"# Access an individual Measurement Set data tree of the processing set\n",
"msv4_name = \"my_msv4\"\n",
"ms_xdt = ps_xdt[msv4_name]\n",
"\n",
"# Access the correlated dataset and antenna dataset\n",
"cor_xds = ms_xdt.ds\n",
"antenna_xds = ms_xdt.antenna_xds.ds\n",
"```\n",
"\n",
"Available accessors:\n",
"\n",
"- ps_xdt.xr_ps (as processing set accessor)\n",
"\n",
" - summary # returns a data frame \n",
"\n",
" - get_max_dims\n",
"\n",
" - get_freq_axis\n",
"\n",
" - sel # based on summary data frame\n",
"\n",
" - get_combined_field_and_source_xds\n",
"\n",
" - plot_phase_centers \n",
"\n",
" - get_combined_antenna_xds\n",
"\n",
" - plot_antenna_positions \n",
"\n",
" - ...\n",
"\n",
"- ms_xdt.xr_ms (as measurement set accesor)\n",
"\n",
" - sel # allows for data group selection\n",
"\n",
" - get_field_and_source_xds\n",
"\n",
" - get_partition_info\n",
"\n",
"All these accessors will be used during the tutorial. Figure 2 gives an schematic overview of the PS and MS data trees.\n",
"\n",
"\n",
"
\n",
" \n",
" \n",
" Figure 2: overview of the PS and MS data trees\n",
" \n",
" \n",
"
\n"
]
},
{
"cell_type": "markdown",
"id": "dc536ba0-5317-4ef1-b3be-1b92a50be58e",
"metadata": {},
"source": [
"## Preparation"
]
},
{
"cell_type": "markdown",
"id": "604007db-1708-487d-99b5-bff82bf3d28b",
"metadata": {},
"source": [
"### Import XRADIO"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b83acc82-3f3d-4c92-aa5d-f25bb01ff8f7",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: xradio in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (1.1.3)\n",
"Requirement already satisfied: xarray in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from xradio) (2026.4.0)\n",
"Requirement already satisfied: numpy>=1.26 in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from xarray->xradio) (2.4.3)\n",
"Requirement already satisfied: packaging>=24.2 in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from xarray->xradio) (26.1)\n",
"Requirement already satisfied: pandas>=2.2 in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from xarray->xradio) (3.0.2)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from pandas>=2.2->xarray->xradio) (2.9.0.post0)\n",
"Requirement already satisfied: six>=1.5 in /Users/vdesouza/miniforge3/envs/xradio-dev/lib/python3.13/site-packages (from python-dateutil>=2.8.2->pandas>=2.2->xarray->xradio) (1.17.0)\n",
"Using xradio version 1.1.3\n"
]
}
],
"source": [
"from importlib.metadata import version\n",
"import os\n",
"\n",
"try:\n",
" os.system(\"pip install --upgrade xradio\")\n",
"\n",
" import xradio\n",
"\n",
" print(\"Using xradio version\", version(\"xradio\"))\n",
"\n",
"except ImportError as exc:\n",
" print(f\"Could not import xradio: {exc}\")"
]
},
{
"cell_type": "markdown",
"id": "f1466b69-66a5-429b-a3d0-f19c7bdd3bfb",
"metadata": {},
"source": [
"### Download example MSv2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "27022146-6bc9-47fd-be7d-b3dc71b1b7ca",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[38;2;128;05;128m2026-04-20 15:21:59,999\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m toolviper: \u001b[0m Initializing download... \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,000\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m toolviper: \u001b[0m File already exists: /Users/vdesouza/work/xradio/docs/source/measurement_set/tutorials/Antennae_North.cal.lsrk.split.ms \n"
]
}
],
"source": [
"import toolviper\n",
"\n",
"toolviper.utils.data.download(file=\"Antennae_North.cal.lsrk.split.ms\")"
]
},
{
"cell_type": "markdown",
"id": "1fda9e81-c44f-4fd3-a582-82580f3b3133",
"metadata": {},
"source": [
"## Processing Set"
]
},
{
"cell_type": "markdown",
"id": "63473e60-615c-4249-b40d-f6068790b47e",
"metadata": {},
"source": [
"### Convert MSv2 to Processing Set Xarray Data Tree (ps_xdt)"
]
},
{
"cell_type": "markdown",
"id": "d0576106-11d2-44bc-a588-d55b0357afa7",
"metadata": {},
"source": [
"Before running the conversion function we can get an estimate of the resources that will be needed:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "85c49f92-da66-4ad5-8c85-d3a75a5902ff",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,281\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m toolviper: \u001b[0m Updated partition scheme used: ['DATA_DESC_ID', 'OBS_MODE', 'OBSERVATION_ID'] \n"
]
},
{
"data": {
"text/plain": [
"(0.010820237919688225, 4, 1)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from xradio.measurement_set import estimate_conversion_memory_and_cores\n",
"\n",
"msv2_name = \"Antennae_North.cal.lsrk.split.ms\"\n",
"mem_estimate, max_reasonable_cores, suggested_cores = estimate_conversion_memory_and_cores(msv2_name)\n",
"mem_estimate, max_reasonable_cores, suggested_cores"
]
},
{
"cell_type": "markdown",
"id": "6eba83b6-5d53-46d7-8e84-bf94759fb60e",
"metadata": {},
"source": [
"The function used to estimate resources gives:\n",
"- an estimate of memory required in GiB,\n",
"- a maximum \"reasonable\" number of cores to use when converting in parallel, which is the number of partitions or MSv4s in the output processing set,\n",
"- and a suggested number of cores to use, as a rule of thumb the maximum / 4.\n",
"\n",
"If we want to run the conversion in parallel, using Dask, we can initialize a \"VIPER\" client. In this example we use a local Dask client with the suggested number of cores = Dask workers:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7febe111-0000-4c51-bb8c-33efe13fd934",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,314\u001b[0m] \u001b[38;2;255;160;0m WARNING\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m It is recommended that the local cache directory be set using the \u001b[38;2;50;50;205mdask_local_dir\u001b[0m parameter. \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,856\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m Client \n"
]
}
],
"source": [
"do_parallel = True\n",
"if do_parallel:\n",
" from toolviper import dask\n",
" viper_client = toolviper.dask.local_client(cores=suggested_cores)"
]
},
{
"cell_type": "markdown",
"id": "e87e00a3-c04d-4e5c-a9f7-b5e2ad1c9b6b",
"metadata": {},
"source": [
"Convert the example MeasurementSet v2 to Processing Set:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "fcd710fb-c2e0-4ff6-8e03-6c11673eec09",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,905\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m Updated partition scheme used: ['DATA_DESC_ID', 'OBS_MODE', 'OBSERVATION_ID'] \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,906\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m Number of partitions: 4 \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,907\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m OBSERVATION_ID [0], DDI [0], STATE [23, 24, 25, 30, 31, 32, 33, 34, 37], FIELD [0, 1, 2], SCAN [9, 17, 21, 25], EPHEMERIS [None] \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,907\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m OBSERVATION_ID [1], DDI [0], STATE [23, 24, 25, 30, 31, 32, 33, 34, 37], FIELD [0, 1, 2], SCAN [26, 34, 38, 42], EPHEMERIS [None] \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,907\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m OBSERVATION_ID [2], DDI [0], STATE [32, 33, 34], FIELD [0, 1, 2], SCAN [43], EPHEMERIS [None] \n",
"[\u001b[38;2;128;05;128m2026-04-20 15:22:00,908\u001b[0m] \u001b[38;2;50;50;205m INFO\u001b[0m\u001b[38;2;112;128;144m client: \u001b[0m OBSERVATION_ID [3], DDI [0], STATE [39, 40, 41, 46, 47, 48, 49, 50, 53], FIELD [0, 1, 2], SCAN [48, 56, 60, 64], EPHEMERIS [None] \n"
]
}
],
"source": [
"from xradio.measurement_set import convert_msv2_to_processing_set\n",
"if do_parallel:\n",
" parallel_mode = \"partition\"\n",
"else:\n",
" parallel_mode = \"none\"\n",
"\n",
"convert_out = \"Antennae_North.cal.lsrk.split.ps.zarr\"\n",
"convert_msv2_to_processing_set(\n",
" in_file=msv2_name,\n",
" out_file=convert_out,\n",
" persistence_mode='w',\n",
" parallel_mode=parallel_mode,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "e69936df-63bd-4a49-adeb-8fba6bc572ff",
"metadata": {},
"source": [
"### Processing Set Xarray Data Tree (ps_xdt)\n",
"\n",
"A `ps_xdt` (Processing Set DataTree) can be lazily opened using either [`xradio.measurement_set.open_processing_set`](https://casangi.github.io/xradio/api/xradio.measurement_set.html#xradio.measurement_set.open_processing_set) or [`xarray.open_datatree`](https://docs.xarray.dev/en/latest/generated/xarray.open_datatree.html). When lazily opened, none of the data variables within the datasets are immediately loaded into memory. Instead, they are represented by lazy Dask arrays. The `open_processing_set` function provided by `xradio.measurement_set` exists to support future storage formats that are not natively handled by xarray. For example, telescopes such as ALMA and the VLA produce data in the ASDM (Astronomical Science Data Model) format, which may require custom loading and conversion logic. By using `open_processing_set`, XRADIO ensures flexibility and extensibility for supporting complex and evolving data formats beyond the scope of standard xarray I/O. \n",
"\n",
"The Processing Set \"Antennae_North.cal.lsrk.split.ps.zarr\" consists of 12 Measurement Sets v4 (`ms_xdt`) which can be seen by clicking on the Groups dropdown arrow."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "02d150f5-f513-4ab8-a2e2-9051d6c79d25",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.436598+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.209379+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.328266+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.436598+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.209379+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.328266+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}, 'corrected': {'correlated_data': 'VISIBILITY_CORRECTED', 'flag': 'FLAG', 'weight': 'WEIGHT_IMAGING', 'uvw': 'UVW', 'field_and_source': 'field_and_source_base_xds', 'description': 'New data group with corrected data', 'date': '2026-04-20T21:22:01.540275+00:00'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
{'base': {'correlated_data': 'VISIBILITY', 'date': '2026-04-20T21:22:01.540275+00:00', 'description': "Data group derived from the data column 'VISIBILITY' of an MSv2 converted to MSv4", 'field_and_source': 'field_and_source_base_xds', 'flag': 'FLAG', 'uvw': 'UVW', 'weight': 'WEIGHT'}}
![\"fig1_ps_vs_mms\"\n",](\"_images/diagram_schema_layout_PS_and_MMS.png\"\n",)
\n",
" ![\"fig2_ps_ms_datatree\"\n",](\"_images/xradio_datatree_ps_ms_diagram.png\"\n",)
\n",
"