Open main menu

Optical music recognition

  (Redirected from Music OCR)

Optical music recognition (OMR) is about computers converting bitmap images of conventional sheet music into machine-readable file formats such as MIDI or MusicXML.[1] There are several approaches to the task, and there are several applications.

In the past it has, misleadingly, also been called Music OCR. Due to significant differences, this term should no longer be used.[2]

HistoryEdit

 
First published digital scan of music scores by David Prerau in 1971

Optical music recognition of printed sheet music started in the late 1960s at MIT when the first image scanners became affordable for research institutes.[3][4][5] Due to the limited memory of early computers, the first attempts were limited to only a few measures of music.

In 1984, a Japanese research group from Waseda University developed a specialized robot, called WABOT (WAseda roBOT), which was capable of reading the music sheet in front of it and accompanying a singer on an electric organ.[6][7]

Early research in OMR was conducted by Ichiro Fujinaga, Nicholas Carter, Kia Ng, David Bainbridge, and Tim Bell. These researchers developed many of the techniques that are still being used today.

The first commercial OMR application, MIDISCAN (now SmartScore), was released in 1991 by Musitek Corporation.

The availability of smartphones with good cameras and sufficient computational power, paved the way to mobile solutions where the user takes a picture with the smartphone and the device directly processes the image.

Relation to other fieldsEdit

 
Relation of optical music recognition to other fields of research

Optical music recognition relates to other fields of research, including computer vision, document analysis, and music information retrieval. It is relevant for practicing musicians and composers that could use OMR systems as a means to enter music into the computer and thus ease the process of composing, transcribing, and editing music. In a library, an OMR system could make music scores searchable[8] and for musicologists it would allow to conduct quantitative musicological studies at scale.[9]

Approaches to OMREdit

 
Excerpt of Nocturne Op. 15, no. 2, by Frédéric Chopin – challenges encountered in optical music recognition

The process of recognizing music scores is typically broken down into smaller steps that are handled with specialized pattern recognition algorithms.

Many competing approaches have been proposed with most of them sharing a pipeline architecture, where each step in this pipeline performs a certain operation, such as detecting and removing staff lines before moving on to the next stage. A common problem with that approach is that errors and artifacts that were made in one stage are propagated through the system and can heavily affect the performance. For example, if the staff line detection stage fails to correctly identify the existence of the music staffs, subsequent steps will probably ignore that region of the image, leading to missing information in the output.

Optical music recognition is frequently underestimated due to the seemingly easy nature of the problem: If provided with a perfect scan of typeset music, the visual recognition can be solved with a sequence of fairly simple algorithms, such as projections and template matching. However, the process gets significantly harder for poor scans or handwritten music, which many systems fail to recognize altogether. And even if all symbols would have been detected perfectly, it is still challenging to recover the musical semantics due to ambiguities and frequent violations of the rules of music notation (see the example of Chopin’s Nocturne). Donald Byrd and Jakob Simonsen argue that OMR is difficult because modern music notation is extremely complex.[10]

Donald Byrd also collected a number of interesting examples[11] as well as extreme examples[12] of music notation that demonstrate the sheer complexity of music notation.

Outputs of OMR systemsEdit

Typical applications for OMR systems include the creation of an audible version of the music score (referred to as replayability). A common way to create such a version is by generating a MIDI file, which can be synthesised into an audio file. MIDI files, though, are not capable of storing engraving information (how the notes were laid out) or enharmonic spelling.

If the music scores are recognized with the goal of human readability (referred to as reprintability), the structured encoding has to be recovered, which includes precise information on the layout and engraving. Suitable formats to store this information include MEI and MusicXML.

General framework (2001)Edit

 
Optical Music Recognition Architecture by Bainbridge and Bell (2001)

In 2001, David Bainbridge and Tim Bell published their work on the challenges of OMR, where they reviewed previous research and extracted a general framework for OMR.[13] Their framework has been used by many systems developed after 2001. The framework has four distinct stages with a heavy emphasis on the visual detection of objects. They noticed that the reconstruction of the musical semantics was often omitted from published articles because the used operations were specific to the output format.

Refined framework (2012)Edit

 
The general framework for optical music recognition proposed by Ana Rebelo et al. in 2012

In 2012, Ana Rebelo et al. surveyed techniques for optical music recognition.[14] They categorized the published research and refined the OMR pipeline into the four stages: Preprocessing, Music symbols recognition, Musical notation reconstruction and Final representation construction. This framework became the de-facto standard for OMR and is still being used today (although sometimes with slightly different terminology). For each block, they give an overview of techniques that are used to tackle that problem. This publication is the most cited paper on OMR research as of 2019.

Deep learning (since 2016)Edit

With the advent of deep learning, many computer vision problems have shifted from imperative programming with hand-crafted heuristics and feature engineering towards machine learning. In optical music recognition, the staff processing stage,[15] the music object detection stage,[16][17][18][19] as well as the music notation reconstruction stage[20] have seen successful attempts to solve them with deep learning.

Even completely new approaches have been proposed, including solving OMR in an end-to-end fashion with sequence-to-sequence models, that take an image of music scores and directly produce the recognized music in a simplified format.[21][22][23][24]

Notable scientific projectsEdit

Staff removal challengeEdit

For systems that were developed before 2016, staff detection and removal posed a significant obstacle. A scientific competition was organized to improve the state of the art and advance the field.[25] Due to excellent results and modern techniques that made the staff removal stage obsolete, this competition was discontinued.

However, the freely available CVC-MUSCIMA dataset that was developed for this challenge is still highly relevant for OMR research as it contains 1000 high-quality images of handwritten music scores, transcribed by 50 different musicians. It has been further extended into the MUSCIMA++ dataset, which contains detailed annotations for 140 out of 1000 pages.

SIMSSAEdit

The Single Interface for Music Score Searching and Analysis project (SIMSSA)[26] is probably the largest project that attempts to teach computers to recognize musical scores and make them accessible. Several sub-projects have already been successfully completed, including the Liber Usualis[27] and Cantus Ultimus.[28]

TROMPAEdit

Towards Richer Online Music Public-domain Archives (TROMPA) is an international research project, sponsored by the European Union that investigates how to make public-domain digital music resources more accessible.[29]

DatasetsEdit

The development of OMR systems benefits from test datasets of sufficient size and diversity to ensure the system being developed works under various conditions. However, due to legal reasons and potential copyright violations, it is challenging to compile and publish such a dataset. The most notable datasets for OMR are referenced and summarized by the OMR Datasets project[30] and include the CVC-MUSCIMA,[31] MUSCIMA++,[32] DeepScores,[33] PrIMuS,[34] HOMUS,[35] and SEILS dataset,[36] as well as the Universal Music Symbol Collection.[37]

SoftwareEdit

Academic and open-source softwareEdit

Many OMR projects have been realized in academia, but only a few of them reached a mature state and were successfully deployed to users. These systems are:

Commercial softwareEdit

Most of the commercial desktop applications that were developed in the last 20 years have been shut down again due to the lack of commercial success, leaving only a few vendors that are still developing, maintaining, and selling OMR products. Some of these products claim extremely high recognition rates with up to 100% accuracy [45][46] but fail to disclose how those numbers were obtained, making it nearly impossible to verify them and compare different OMR systems. Apart from the desktop applications, a range of mobile applications have emerged as well, but received mixed reviews on the Google Play store and were probably discontinued (or at least did not receive any update since 2017).[47][48][49] A range of OMR apps can also be found for iPhone and iPad devices in the Apple Store.[50][51][52][53][54]

  • capella-scan[55]
  • ForteScan Light by Fortenotation[56] now Scan Score[57]
  • MIDI-Connections Scan by MIDI-Connections[58]
  • MP Scan by Braeburn.[59] Uses SharpEye SDK.
  • NoteScan bundled with Nightingale[60]
  • OMeR (Optical Music easy Reader) Add-on for Harmony Assistant and Melody Assistant: Myriad Software[61] (ShareWare)
  • PDFtoMusic[62]
  • PhotoScore by Neuratron.[46] The Light version of PhotoScore is used in Sibelius. PhotoScore uses the SharpEye SDK.
  • Scorscan by npcImaging.[63] Based on SightReader(?)
  • SharpEye by Visiv[64]
    • VivaldiScan (same as SharpEye)[65]
  • SmartScore by Musitek.[66] Formerly packaged as "MIDISCAN". (SmartScore Lite is used in Finale).

See alsoEdit

  • Music information retrieval (MIR) is the broader problem of retrieving music information from media including music scores and audio.
  • Optical character recognition (OCR) is the recognition of text which can be applied to document retrieval, analogously to OMR and MIR. However, a complete OMR system must faithfully represent text that is present in music scores, so OMR is in fact a superset of OCR.[13]

ReferencesEdit

  1. ^ Pacha, Alexander (2019). Self-Learning Optical Music Recognition (PhD). TU Wien, Austria. doi:10.13140/RG.2.2.18467.40484.
  2. ^ Calvo-Zaragoza, Jorge; Hajič, Jan jr.; Pacha, Alexander (2019). "Understanding Optical Music Recognition". arXiv:1908.03608 [cs.CV].
  3. ^ Fujinaga, Ichiro (2018). The History of OMR on YouTube
  4. ^ Pruslin, Dennis Howard (1966). Automatic Recognition of Sheet Music (PhD). Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.
  5. ^ Prerau, David S. (1971). Computer pattern recognition of printed music. Fall Joint Computer Conference. pp. 153–162.
  6. ^ "WABOT – WAseda roBOT". Waseda University Humanoid. Retrieved July 14, 2019.
  7. ^ "Wabot's entry in the IEEE collection of Robots". IEEE. Retrieved July 14, 2019.
  8. ^ Laplante, Audrey; Fujinaga, Ichiro (2016). Digitizing Musical Scores: Challenges and Opportunities for Libraries. 3rd International Workshop on Digital Libraries for Musicology. pp. 45–48.
  9. ^ Hajič, Jan jr.; Kolárová, Marta; Pacha, Alexander; Calvo-Zaragoza, Jorge (2018). How Current Optical Music Recognition Systems Are Becoming Useful for Digital Libraries. 5th International Conference on Digital Libraries for Musicology. Paris, France. pp. 57–61.
  10. ^ Byrd, Donald; Simonsen, Jakob Grue (2015). "Towards a Standard Testbed for Optical Music Recognition: Definitions, Metrics, and Page Images". Journal of New Music Research. 44 (3): 169–195. doi:10.1080/09298215.2015.1045424.
  11. ^ "Gallery of Interesting Music Notation". Donald Byrd. Retrieved July 14, 2019.
  12. ^ "Extremes of Conventional Music Notation". Donald Byrd. Retrieved July 14, 2019.
  13. ^ a b Bainbridge, David; Bell, Tim (2001). "The challenge of optical music recognition". Computers and the Humanities. 35 (2): 95–121. doi:10.1023/A:1002485918032. Retrieved 23 February 2017.
  14. ^ Rebelo, Ana; Fujinaga, Ichiro; Paszkiewicz, Filipe; Marcal, Andre R.S.; Guedes, Carlos; Cardoso, Jamie dos Santos (2012). "Optical music recognition: state-of-the-art and open issues" (PDF). International Journal of Multimedia Information Retrieval. 1 (3): 173–190. doi:10.1007/s13735-012-0004-6.
  15. ^ Castellanos, Fancisco J.; Calvo-Zaragoza, Jorge; Vigliensoni, Gabriel; Fujinaga, Ichiro (2018). Document Analysis of Music Score Images with Selectional Auto-Encoders (PDF). 19th International Society for Music Information Retrieval Conference. Paris, France. pp. 256–263.
  16. ^ Tuggener, Lukas; Elezi, Ismail; Schmidhuber, Jürgen; Stadelmann, Thilo (2018). Deep Watershed Detector for Music Object Recognition (PDF). 19th International Society for Music Information Retrieval Conference. Paris, France. pp. 271–278.
  17. ^ Hajič, Jan jr.; Dorfer, Matthias; Widmer, Gerhard; Pecina, Pavel (2018). Towards Full-Pipeline Handwritten OMR with Musical Symbol Detection by U-Nets (PDF). 19th International Society for Music Information Retrieval Conference. Paris, France. pp. 225–232.
  18. ^ Pacha, Alexander; Hajič, Jan jr.; Calvo-Zaragoza, Jorge (2018). "A Baseline for General Music Object Detection with Deep Learning". Applied Sciences. 8 (9): 1488–1508. doi:10.3390/app8091488.
  19. ^ Pacha, Alexander; Choi, Kwon-Young; Coüasnon, Bertrand; Ricquebourg, Yann; Zanibbi, Richard; Eidenberger, Horst (2018). Handwritten Music Object Detection: Open Issues and Baseline Results (PDF). 13th International Workshop on Document Analysis Systems. pp. 163–168. doi:10.1109/DAS.2018.51.
  20. ^ Pacha, Alexander; Calvo-Zaragoza, Jorge; Hajič, Jan jr. (2019). Learning Notation Graph Construction for Full-Pipeline Optical Music Recognition. 20th International Society for Music Information Retrieval Conference (in press).
  21. ^ van der Wel, Eelco; Ullrich, Karen (2017). Optical Music Recognition with Convolutional Sequence-to-Sequence Models (PDF). 18th International Society for Music Information Retrieval Conference. Suzhou, China.
  22. ^ Calvo-Zaragoza, Jorge; Rizo, David (2018). "End-to-End Neural Optical Music Recognition of Monophonic Scores". Applied Sciences. 8 (4): 606. doi:10.3390/app8040606.
  23. ^ Baró, Arnau; Riba, Pau; Calvo-Zaragoza, Jorge; Fornés, Alicia (2017). Optical Music Recognition by Recurrent Neural Networks. 14th International Conference on Document Analysis and Recognition. pp. 25–26. doi:10.1109/ICDAR.2017.260.
  24. ^ Baró, Arnaud; Riba, Pau; Calvo-Zaragoza, Jorge; Fornés, Alicia (2019). "From Optical Music Recognition to Handwritten Music Recognition: A baseline". Pattern Recognition Letters. 123: 1–8. doi:10.1016/j.patrec.2019.02.029.
  25. ^ Fornés, Alicia; Dutta, Anjan; Gordo, Albert; Lladós, Josep (2013). "The 2012 Music Scores Competitions: Staff Removal and Writer Identification". Graphics Recognition. New Trends and Challenges. Lecture Notes in Computer Science. Springer. 7423: 173–186. doi:10.1007/978-3-642-36824-0_17. ISBN 978-3-642-36823-3.
  26. ^ "The SIMSSA project website". McGill University. Retrieved July 14, 2019.
  27. ^ "The Liber Usualis project website". McGill University. Retrieved July 14, 2019.
  28. ^ "The Cantus Ultimus project website". McGill University. Retrieved July 14, 2019.
  29. ^ "The TROMPA project website". Trompa consortium. Retrieved July 14, 2019.
  30. ^ "The OMR Datasets Project (Github Repository)". Pacha, Alexander. Retrieved July 14, 2019.
  31. ^ Fornés, Alicia; Dutta, Anjan; Gordo, Albert; Lladós, Josep (2012). "CVC-MUSCIMA: A Ground-truth of Handwritten Music Score Images for Writer Identification and Staff Removal". International Journal on Document Analysis and Recognition. 15 (3): 243–251. doi:10.1007/s10032-011-0168-2.
  32. ^ Hajič, Jan jr.; Pecina, Pavel (2017). The MUSCIMA++ Dataset for Handwritten Optical Music Recognition. 14th International Conference on Document Analysis and Recognition. Kyoto, Japan. pp. 39–46. doi:10.1109/ICDAR.2017.16.
  33. ^ Tuggener, Lukas; Elezi, Ismail; Schmidhuber, Jürgen; Pelillo, Marcello; Stadelmann, Thilo (2018). DeepScores – A Dataset for Segmentation, Detection and Classification of Tiny Objects. 24th International Conference on Pattern Recognition. Beijing, China. doi:10.21256/zhaw-4255.
  34. ^ Calvo-Zaragoza, Jorge; Rizo, David (2018). Camera-PrIMuS: Neural End-to-End Optical Music Recognition on Realistic Monophonic Scores (PDF). 19th International Society for Music Information Retrieval Conference. Paris, France. pp. 248–255.
  35. ^ Calvo-Zaragoza, Jorge; Oncina, Jose (2014). Recognition of Pen-Based Music Notation: The HOMUS Dataset. 22nd International Conference on Pattern Recognition. pp. 3038–3043. doi:10.1109/ICPR.2014.524.
  36. ^ Parada-Cabaleiro, Emilia; Batliner, Anton; Baird, Alice; Schuller, Björn (2017). The SEILS Dataset: Symbolically Encoded Scores in Modern-Early Notation for Computational Musicology (PDF). 18th International Society for Music Information Retrieval Conference. Suzhou, China. pp. 575–581.
  37. ^ Pacha, Alexander; Eidenberger, Horst (2017). Towards a Universal Music Symbol Classifier. 14th International Conference on Document Analysis and Recognition. Kyoto, Japan. pp. 35–36. doi:10.1109/ICDAR.2017.265.
  38. ^ Aruspix
  39. ^ Audiveris
  40. ^ CANTOR
  41. ^ Gamera
  42. ^ Coüasnon, Bertrand (2001). DMOS: a generic document recognition method, application to an automatic generator of musical scores, mathematical formulae and table structures recognition systems. Sixth International Conference on Document Analysis and Recognition. pp. 215–220. doi:10.1109/ICDAR.2001.953786.
  43. ^ OpenOMR
  44. ^ Rodan
  45. ^ Info on the accuracy of capella-scan
  46. ^ a b PhotoScore Ultimate 7
  47. ^ PlayScore Pro
  48. ^ iSeeNotes
  49. ^ NotateMe Now
  50. ^ MusicPal
  51. ^ Sheet Music Scanner
  52. ^ PlayScore 2
  53. ^ Notation Scanner - Music OCR
  54. ^ Komp Create
  55. ^ Info capella-scan
  56. ^ FORTE Scan Light Archived 2013-09-22 at the Wayback Machine
  57. ^ Scan Score
  58. ^ MIDI-Connections SCAN 2.0 Archived 2013-12-20 at the Wayback Machine
  59. ^ Music Publisher Scanning Edition Archived 2013-04-13 at the Wayback Machine
  60. ^ NoteScan
  61. ^ OMeR
  62. ^ PDFtoMusic
  63. ^ ScorScan
  64. ^ SharpEye
  65. ^ VivaldiScan Archived 2005-12-24 at the Wayback Machine
  66. ^ SmartScore Archived 2012-04-17 at the Wayback Machine

External linksEdit