Quantification of intratumoural heterogeneity in mice and patients via machine

Sep 03, 2023

Nature Biomedical Engineering (2023)Cite this article

2 Altmetric

Metrics details

In oncology, intratumoural heterogeneity is closely linked with the efficacy of therapy, and can be partially characterized via tumour biopsies. Here we show that intratumoural heterogeneity can be characterized spatially via phenotype-specific, multi-view learning classifiers trained with data from dynamic positron emission tomography (PET) and multiparametric magnetic resonance imaging (MRI). Classifiers trained with PET–MRI data from mice with subcutaneous colon cancer quantified phenotypic changes resulting from an apoptosis-inducing targeted therapeutic and provided biologically relevant probability maps of tumour-tissue subtypes. When applied to retrospective PET–MRI data of patients with liver metastases from colorectal cancer, the trained classifiers characterized intratumoural tissue subregions in agreement with tumour histology. The spatial characterization of intratumoural heterogeneity in mice and patients via multimodal, multiparametric imaging aided by machine-learning may facilitate applications in precision oncology.

This is a preview of subscription content, access via your institution

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

$29.99 / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

$79.00 per year

only $6.58 per issue

Rent or buy this article

Get just this article for as long as you need it

$39.95

Prices may be subject to local taxes which are calculated during checkout

The main data supporting the results in this study are available within the paper and its Supplementary Information. The preclinical data are available from the corresponding author on reasonable request. Clinical data cannot be shared owing to patient-confidentiality obligations.

Wherever applicable, default methods from MATLAB were used for the machine-learning analyses. Custom code for visualization and data processing is available from the corresponding author on request.

Eisenhauer, E. A. et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45, 228–247 (2009).

Article CAS PubMed Google Scholar

O’Connor, J. P. B. et al. Imaging biomarker roadmap for cancer studies. Nat. Rev. Clin. Oncol. 14, 169–186 (2016).

Article PubMed PubMed Central Google Scholar

Judenhofer, M. S. et al. Simultaneous PET–MRI: a new approach for functional and morphological imaging. Nat. Med. 14, 459–465 (2008).

Article CAS PubMed Google Scholar

Disselhorst, J. A., Bezrukov, I., Kolb, A., Parl, C. & Pichler, B. J. Principles of PET/MR imaging. J. Nucl. Med. 55, 2S–10S (2014).

Article PubMed Google Scholar

Bailey, D. L. et al. Combined PET/MR: the real work has just started. Summary report of the Third International Workshop on PET/MR Imaging; February 17–21, 2014, Tübingen, Germany. Mol. Imaging Biol. 17, 297–312 (2015).

Article CAS PubMed PubMed Central Google Scholar

Gillies, R. J. & Beyer, T. PET and MRI: is the whole greater than the sum of its parts? Cancer Res. 76, 6163–6166 (2016).

Article CAS PubMed PubMed Central Google Scholar

Schmitz, J. et al. Decoding intratumoral heterogeneity of breast cancer by multiparametric in vivo imaging: a translational study. Cancer Res. 76, 5512–5522 (2016).

Article CAS PubMed PubMed Central Google Scholar

O’Connor, J. P. B. et al. Imaging intratumor heterogeneity: role in therapy response, resistance, and clinical outcome. Clin. Cancer Res. 21, 249–257 (2015).

Article PubMed Google Scholar

Napel, S., Mu, W., Jardim-Perassi, B. V., Aerts, H. J. W. L. & Gillies, R. J. Quantitative imaging of cancer in the postgenomic era: radio(geno)mics, deep learning, and habitats. Cancer 124, 4633–4649 (2018).

Heinzmann, K., Carter, L. M., Lewis, J. S. & Aboagye, E. O. Multiplexed imaging for diagnosis and therapy. Nat. Biomed. Eng 1, 697–713 (2017).

Article PubMed Google Scholar

Junttila, M. R. & de Sauvage, F. J. Influence of tumour micro-environment heterogeneity on therapeutic response. Nature 501, 346–354 (2013).

Article CAS PubMed Google Scholar

Schmidt, H. et al. Correlation of simultaneously acquired diffusion-weighted imaging and 2-deoxy-[18F] fluoro-2-D-glucose positron emission tomography of pulmonary lesions in a dedicated whole-body magnetic resonance/positron emission tomography system. Invest. Radiol. 48, 247–255 (2013).

Article PubMed Google Scholar

Divine, M. R. et al. A population-based gaussian mixture model incorporating 18F-FDG PET and diffusion-weighted MRI quantifies tumor tissue classes. J. Nucl. Med. 57, 473–479 (2016).

Article CAS PubMed Google Scholar

Kim, J., Ryu, S. Y., Lee, S. H., Lee, H. Y. & Park, H. Clustering approach to identify intratumour heterogeneity combining FDG PET and diffusion-weighted MRI in lung adenocarcinoma. Eur. Radiol. 29, 468–475 (2019).

Article PubMed Google Scholar

Stoyanova, R. et al. Association of multiparametric MRI quantitative imaging features with prostate cancer gene expression in MRI-targeted prostate biopsies. Oncotarget 7, 53362–53376 (2016).

Article PubMed PubMed Central Google Scholar

Katiyar, P. et al. A novel unsupervised segmentation approach quantifies tumor tissue populations using multiparametric MRI: first results with histological validation. Mol. Imaging Biol. 19, 391–397 (2017).

Article PubMed Google Scholar

Katiyar, P. et al. Spectral clustering predicts tumor tissue heterogeneity using dynamic 18 F-FDG PET: a complement to the standard compartmental modeling approach. J. Nucl. Med. 58, 651–657 (2017).

Article CAS PubMed Google Scholar

Carano, R. A. D. et al. Quantification of tumor tissue populations by multispectral analysis. Magn. Reson. Med. 51, 542–551 (2004).

Article PubMed Google Scholar

Berry, L. R. et al. Quantification of viable tumor microvascular characteristics by multispectral analysis. Magn. Reson. Med. 60, 64–72 (2008).

Article PubMed Google Scholar

Barck, K. H. et al. Viable tumor tissue detection in murine metastatic breast cancer by whole-body MRI and multispectral analysis. Magn. Reson. Med. 62, 1423–1430 (2009).

Article PubMed Google Scholar

Henning, E. C., Azuma, C., Sotak, C. H. & Helmer, K. G. Multispectral quantification of tissue types in a RIF-1 tumor model with histological validation. Part I. Magn. Reson. Med. 57, 501–512 (2007).

Article PubMed Google Scholar

Schölkopf, B. Artificial intelligence: learning to see and act. Nature 518, 486–487 (2015).

Article PubMed Google Scholar

de Vries, E. G. E. et al. Integrating molecular nuclear imaging in clinical research to improve anticancer therapy. Nat. Rev. Clin. Oncol. 16, 241–255 (2019).

Article PubMed Google Scholar

Siegemund, M. et al. Superior antitumoral activity of dimerized targeted single-chain TRAIL fusion proteins under retention of tumor selectivity. Cell Death Dis. 3, e295 (2012).

Article CAS PubMed PubMed Central Google Scholar

Gillies, R. J., Kinahan, P. E. & Hricak, H. Radiomics: images are more than pictures, they are data. Radiology 278, 563–577 (2016).

Article PubMed Google Scholar

Aerts, H. J. W. L. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 5, 4006 (2014).

Article CAS PubMed Google Scholar

Kumar, V. et al. Radiomics: the process and the challenges. Magn. Reson. Imaging 30, 1234–1248 (2012).

Article PubMed PubMed Central Google Scholar

Stewart, G. D. et al. Sunitinib treatment exacerbates intratumoral heterogeneity in metastatic renal cancer. Clin. Cancer Res. 21, 4212–4223 (2015).

Article CAS PubMed Google Scholar

Lee, B. S. et al. Induced phenotype targeted therapy: radiation-induced apoptosis-targeted chemotherapy. J. Natl Cancer Inst. 107, dju403 (2015).

Article PubMed Google Scholar

Disselhorst, J. A. et al. Linking imaging to omics utilizing image-guided tissue extraction. Proc. Natl Acad. Sci. USA 115, E2980–E2987 (2018).

Article CAS PubMed PubMed Central Google Scholar

Jaffray, D. A. Image-guided radiotherapy: from current concept to future perspectives. Nat. Rev. Clin. Oncol. 9, 688–699 (2012).

Article CAS PubMed Google Scholar

Hynynen, K. MRIgHIFU: a tool for image-guided therapeutics. J. Magn. Reson. Imaging 34, 482–493 (2011).

Article PubMed Google Scholar

Reinke, A. et al. Common limitations of image processing metrics: a picture story. Preprint at https://arxiv.org/abs/2104.05642 (2021).

Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).

Article CAS PubMed Google Scholar

Sequist, L. V. et al. Genotypic and histological evolution of lung cancers acquiring resistance to EGFR inhibitors. Sci. Transl. Med. 3, 75ra26 (2011).

Article PubMed PubMed Central Google Scholar

Hodi, F. S. et al. Evaluation of immune-related response criteria and RECIST v1.1 in patients with advanced melanoma treated with pembrolizumab. J. Clin. Oncol. 34, 1510–1517 (2016).

Article CAS PubMed PubMed Central Google Scholar

Veuthey, T. V., Herrera, G. & Dodero, V. I. Dyes and stains: from molecular structure to histological application. Front. Biosci. 19, 91–112 (2014).

Article CAS Google Scholar

Gown, A. M. & Willingham, M. C. Improved detection of apoptotic cells in archival paraffin sections: immunohistochemistry using antibodies to cleaved caspase 3. J. Histochem. Cytochem. 50, 449–454 (2002).

Article CAS PubMed Google Scholar

Austyn, J. M. & Gordon, S. F4/80, a monoclonal antibody directed specifically against the mouse macrophage. Eur. J. Immunol. 11, 805–815 (1981).

Article CAS PubMed Google Scholar

Dubuisson, M.-P. & Jain, A. K. A modified Hausdorff distance for object matching. In Proc. 12th International Conference on Pattern Recognition 566–568 (IEEE, 1994).

Dice, L. R. Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945).

Article Google Scholar

von Luxburg, U. A tutorial on spectral clustering. Stat. Comput. 17, 395–416 (2007).

Article Google Scholar

Kumar, A. & Daumé, H. A co-training approach for multi-view spectral clustering. In Proc. 28th International Conference on Machine Learning (ICML-11) 393–400 (ACM, 2011).

Strobl, C., Boulesteix, A.-L., Zeileis, A. & Hothorn, T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 8, 25 (2007).

Article PubMed PubMed Central Google Scholar

Download references

We thank S. Castaneda Vega from the Werner Siemens Imaging Center for help with image analysis and M. Siegemund from the University of Stuttgart for providing support with Db-scTRAIL therapy. B.J.P. received support for the research described in this study from the European Union Seventh Framework Programme (FP7/2007–2013) under the European Research Council grant agreement number 323196 (ImageLink) and Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy (EXC-2180­390900677). BS is a member of the excellence cluster "Machine Learning in the Sciences" funded by the Deutsche Forschungsgemeinschaft under Germany's Excellence Strategy – EXC number 2064/1 – Project number 390727645.

Werner Siemens Imaging Center, Department of Preclinical Imaging and Radiopharmacy, Eberhard Karls University Tübingen, Tübingen, Germany

Prateek Katiyar, Johannes Schwenck, Mathew R. Divine, Vaibhav Agrawal, Bernd J. Pichler & Jonathan A. Disselhorst

Max Planck Institute for Intelligent Systems, Tübingen, Germany

Prateek Katiyar, Vaibhav Agrawal & Bernhard Schölkopf

Cluster of Excellence iFIT (EXC 2180) ‘Image Guided and Functionally Instructed Tumor Therapies’, Eberhard Karls University Tübingen, Tübingen, Germany

Johannes Schwenck, Sergios Gatidis, Alfred Königsrainer, Leticia Quintanilla-Martinez, Christian la Fougère, Bernhard Schölkopf, Bernd J. Pichler & Jonathan A. Disselhorst

Department of Nuclear Medicine and Clinical Molecular Imaging, Eberhard Karls University Tübingen, Tübingen, Germany

Johannes Schwenck & Christian la Fougère

Institute of Pathology and Neuropathology, Eberhard Karls University Tübingen and Comprehensive Cancer Center, University Hospital Tübingen, Tübingen, Germany

Leonie Frauenfeld, Ursula Kohlhofer & Leticia Quintanilla-Martinez

Department of Radiology, Eberhard Karls University Tübingen, Tübingen, Germany

Sergios Gatidis

Institute of Cell Biology and Immunology, SRCSB, University of Stuttgart, Stuttgart, Germany

Roland Kontermann

Department of General, Visceral and Transplant Surgery, University Hospital Tübingen, Tübingen, Germany

Alfred Königsrainer

German Cancer Consortium (DKTK) and German Cancer Research Center (DKFZ), Heidelberg, Germany

Christian la Fougère & Bernd J. Pichler

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

P.K., J.A.D., M.R.D. and B.J.P. designed the preclinical imaging and ex vivo experiments. J.A.D. and M.R.D. collected the preclinical imaging data. J.S., S.G. and C.l.F. conceptualized and performed the clinical imaging work. L.F., U.K. and L.Q.-M. performed the histopathology work and wrote the pathology findings. R.K. provided the Db-scTRAIL therapy. A.K. performed the surgery. P.K., J.A.D., B.S. and B.J.P. designed the machine-learning algorithmic pipeline. P.K. performed the machine-learning experiments and prepared all figures. P.K., V.A. and J.S. performed the data analysis. P.K., J.S., L.F., U.K., L.Q.-M., J.A.D and B.J.P. wrote the paper. All the authors contributed to the final version of the paper.

Correspondence to Bernd J. Pichler.

P.K. is working at Novartis Institutes for BioMedical Research on artificial intelligence and machine-learning. J.A.D. is working at Siemens Healthineers. B.S. has a spin-off company and is working with several companies on artificial intelligence and machine-learning. B.J.P. has several research collaborations with Siemens, Bruker and larger pharmaceutical companies. However, none of these affiliations are directly linked to this work. The other authors declare no competing interests. R.K. is a named inventor on patents covering the TRAIL technology and a consultant to Immatics, Roche, SunRock and Oncomatryx.

Nature Biomedical Engineering thanks Nathaniel Braman, Faisal Mahmood and Natarajan Raghunand for their contribution to the peer review of this work.

Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

a, Left to right: A-Casp-3 and F4-80 stainings of an exemplary Th-24 tumour. The images in the inset show macrophages present at the border and in the fibrous area of the tumour. b, The MT insets show two non-apoptotic regions of the Th-24 tumour that majorly consist of loose fibrous tissue. c, The H&E inset additionally shows a small necrotic region present at the top of the tumour.

Left to right, pairs of columns: Viable, NF and apoptosis probability maps of the CRLM shown in Fig. 6, predicted by the PET and MRI classifiers. For both the models, the probability values of each voxel across the maps of all phenotypes sum to one.

The Pearson's correlation plot illustrating the relationship between the viable and NF tissue fractions predicted by the a, PET/MRI, b, MRI, and c, PET Random Forest classifiers and the ground truth evaluations made by the pathologist using tumour histology. The correlation coefficients are indicated by the symbol ρ. For each histology slide, the classifier predicted viable, apoptosis and NF tissue fractions sum to 1.0.

Column-wise, left to right: H&E, A-Casp-3 and MT histology of six representative test set tumours, and the corresponding phenotypic maps predicted by the PET/MRI and PET/MRITOP classifiers. The PET/MRITOP model was trained using the combined training data of the eight most relevant PET/MRI features, shown in Fig. 4g. The top, middle and bottom pairs of the tumours belong to the test Control, Th-24 and Th-72 groups, respectively. The phenotypic maps are joint probability maps that were colour-coded based on the tumour tissue class colour map shown in the key. Thus, the trained classifiers probabilistically assigned each voxel in the phenotypic maps to either one of the three phenotypic classes. The colour-coded arrows in the histology indicate the respective phenotypic class. The image in the first inset shows a patch of the Control tumour containing viable tumour tissue and necrosis. The second inset illustrates bundles of viable tumour cells distributed within a largely apoptotic region of the Th-24 tumour. Lastly, the image in the third inset shows the highly complex microenvironment of the Th-72 tumour.

Column-wise, left to right: H&E histology of four CRLM each resected from a different patient, the corresponding phenotypic maps predicted by the PETTOP, MRITOP, PET/MRITOP and PET/MRI classifiers, and the factored probability maps of each phenotype for the PET/MRITOP model. The PET/MRITOP model was trained using the combined training data of the eight most relevant PET/MRI features, shown in Fig. 4g. Whereas the PETTOP and MRITOP models were trained using the relevant features only from the respective modalities. The solid lines partition the presented examples patient-wise and the blue contours in the histology isolate the tumour from the liver tissue. The metastasis in the third row was too large to be processed on a single histology slide, and therefore had to be sectioned into three separate pieces. The dashed black lines in the stitched H&E image show the borders of the separated tumour units. The phenotypic maps are joint probability maps that were colour-coded based on the tumour tissue class colour map shown in the key. Thus, the trained classifiers probabilistically assigned each voxel in the phenotypic maps to either one of the three phenotypic classes. The colour-coded arrows in the histology indicate the respective phenotypic class, whereas the white arrow points towards the tumour region that was falsely classified by all models. All four insets show viable tumour regions within the largely necrotic CRLM. The factored probability maps of the PET/MRI classifier are shown in Fig. 6.

The MSC workflow broadly comprised three analysis stages. The first and second stages utilized spectral clustering to segment the multiparametric MRI and dynamic 18F-FDG PET data, respectively. In the third stage, complementary information from both the previous steps was combined to obtain the PET/MRI maps of intratumoural heterogeneity. WMRI and WPET denote the affinity matrices that yielded the optimal clustering results for the associated modalities. The PET/MRI affinity matrix, WPET/MRI, was obtained using a convex combination of the matrices from the two views, where the parameter α regulated the influence of each view. The optimal parameters in each step were selected based on a grid search to maximize the DSC between the histology and imaging clustering maps.

Supplementary Methods and references.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

Katiyar, P., Schwenck, J., Frauenfeld, L. et al. Quantification of intratumoural heterogeneity in mice and patients via machine-learning models trained on PET–MRI data. Nat. Biomed. Eng (2023). https://doi.org/10.1038/s41551-023-01047-9

Download citation

Received: 05 January 2021

Accepted: 26 April 2023

Published: 05 June 2023

DOI: https://doi.org/10.1038/s41551-023-01047-9

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative