About the project

About the project.

The SHIFT consortium brings together 13 leading research and industrial organisations and SMEs, coordinated by SIMAVI (Romania), with the vision to strengthen the impact of cultural heritage assets. It will increase the appeal of historical artefacts, improving their accessibility and usability for everyone through better content representation, enriched user experiences, inclusive design, training and more engaging business models.

General overview.

SHIFT: MetamorphoSis of cultural Heritage Into augmented hypermedia assets For enhanced accessibiliTy and inclusion supports the adoption of digital transformation strategies and the uptake of tools within the creative and cultural industries (CCI), where progress has been lagging.

Objectives.

The project is strategically conceived to deliver, by advancing beyond the state-of-the-art, a set of loosely coupled technological tools that offers cultural heritage institutions (CHI) the impetus to stimulate growth and communicate new experiences to all citizens (including people with disabilities) by embracing the latest technological innovations in

artificial intelligence
machine learning
multi-modal data processing
digital content transformation methodologies
semantic representation
linguistic analysis of historical records
use of haptics interfaces

Consortium.

SIMAVI SOFTWARE IMAGINATION & VISION SRL (Coordinator)

Romania

Software Imagination & Vision (SIMAVI) is a software company, a partial take over from SIVECO Romania – a large software company established in 1992. SIMAVI has inherited all the assets related to the software development activities, the related experience in the field, the implementation teams, the certificates, and the authorizations held by SIVECO Romania.

SIMAVI continues the outstanding reputation of SIVECO, in international markets, by developing successful projects together with several international companies.

Moreover, SIMAVI has a significant experience and an exceptional track record in R&D&I projects, being involved as technological provider and as coordinator in many European and National research projects.

SIMAVI provides services on the whole life cycle of projects: analysis of user’s requirements, implementation, end-users training and technical assistance, and system maintenance. SIMAVI staff consists of IT specialists covering all stages of projects development: analysts, business consultants, system architects, programmers for different platforms, implementers, testers, DBAs, data analysts, accredited trainers. The solutions developed by SIMAVI are based on successful and mature existing products and custom-made software, operating on various platforms and databases. They are web standard compatible and can be easily integrated with other applications and internal or external databases (Geographical Information Systems, Database Sharing, Business Intelligence Applications, etc).

SIMAVI is the project coordinator leveraging on its experience in managing large-scale R&D H2020 projects. It will also lead to exploitation of project outcomes and contribute towards the platform integration services.

In the SHIFT project SIMAVI it’s responsible:

To design and develop a valid architectural system and implement a functional platform and define integration patterns.
To ensure full functional integration based on the architectural design.
To implement a back-end system that currently supports “content management services” and user management for all actors.
To perform platform and tools wide validation with specific end-user pilot

ANBPR ASOCIATIA NATIONALA A BIBLIOTECARILOR SI BIBLIOTECILOR PUBLICE DIN ROMANIA

Romania

ANBPR is the most representative professional association of librarians in Romania, including over 3,300 members in 41 county branches and has promoted for the last 32 years the concept of a public library system at community level by providing services tailored to the current needs of library users. As an NGO interested in social inclusion of disadvantaged people, ANBPR is committed to widening participation in cultural education through links with local communities having experience in providing modern services. ANBPR is promoting a dialogue amongst main cultural “actors” in rural or urban areas of Romania: librarians, writers, artists, curators, teachers, educators or any other promoter of universal cultural values at local level, using as main `scenes` the Romanian public libraries. Since 1990, ANBPR has aimed to coagulate, in a form of association at national level, the current needs of the librarian profession, to promote a nationwide system of modern public libraries. Through its ANBPR has carried out a wide range of activities with the participation of its members, from national events, occupational standards, public policies, advocacy initiatives and collaboration with different partners from civil society. According to its mission, ANBPR aims to coagulate and give voice to the expectations, needs and interests of those who, through their profession, concerns and duties, support the librarian profession, the development of librarianship and the sciences of information and documentation.

ANBPR, a national association of librarians, will lead the effort on eliciting end-user specification for SHIFT digital tools and facilitate promotion of project outcomes.

AUDEERING GMBH

Germany

audEERING GmbH is involved in developing and licensing software for intelligent audio analysis, in particular speech and sound-scapes. A core software product is the world-class audio analysis toolkit openSMILE, which is capable of extracting over 6k audio features relevant for speech, music, and sound analysis and has been used as a basis for numerous international benchmarks. Combined with machine learning expertise, audEERING delivers highly accurate sound and speech analysis products.

audEERING has been founded late 2012 as a Spin-off of Technische Universität München, Germany (Machine Intelligence and Signal Processing Group) with a globally recognized team that has 20 years of audio intelligence experience. audEERING is based in Gilching near Munich and Berlin, Germany with about 44 salaried employees and about 20 associated freelance data annotators.

audEERING is involved in WP3 with the task to develop a system for Emotional Speech Synthesis. The purpose of this system will be to tell the story of historical artefacts from museums and libraries. The system will have the ability to reveal the underlying emotion discovered in its input text using text Sentiment Analysis and subsequently generate a voice reciting the input text with an equivalent emotion.

BMN UDRUZENJA BALKANSKA MREZA MUZEJA

Bosnia and Herzegovina

The Balkan Museum Network is an independent grassroots Non Governmental Organisation (NGO) that links museum professionals. The strapline of the Balkan Museum Network (BMN) is ‘Join us in joining hands’, an illustration of its grassroots positioning and its commitment to promoting incremental change from the bottom up. Applications of (radical) museum and heritage practice are grounded in the local experience and context of members and activists. As such, the BMN is advocate, connector, actor, agent of change and mentor, meaning that it sits on the margins of official museum structures that remain predominant throughout many of the former socialist states.

BMN continues to provide transformative programmes and international collaborations that significantly strengthen museums at a local level. The key support of the network is providing a safe space for learning and sharing personal experiences that resonate among colleagues across the borders. The BMN’s importance is in creating a forum for the voice of middle managers and professionals.

Role in the project:

5.1 leader: BMN will be involved in the creation of the multi-modal cultural experience organised as Exhibition case 1: 19^th modern days Serbian paintings and modern art, in the Homeland Museum in Knjazevac (Serbia). The objective of this task is to realize an iterative evaluation of the selected tools with the aim to augment the experience of visitors for an exhibition focused on the most significant 19^th century and contemporary Serbian paintings using innovative tools.
5: Monetization leader: Explore monetization strategies, examine both editions’ uptake to finalise a market acceptable pricing scheme. Investigate how to maximize profitability based on early user feedback from Trials.

DBSV DEUTSCHER BLINDEN- UND SEHBEHINDERTENVERBAND EV

Germany

The DBSV (“Deutscher Blinden- und Sehbehindertenverband”) is the German Federation of the Blind and Partially Sighted, the national organisation of visually impaired persons and patients who suffer from a disease leading to sight loss. As self-help organisation, DBSV strives for an inclusive, forward-looking society free of discrimination.

As the umbrella organisation of the different German self-help organisations of the blind and partially sighted, DBSV combines and coordinates efforts of 19 regional associations. Our regional associations represent around 36,000 blind and partially sighted people at the local level, offering comprehensive advice and leisure activities. Thirty-five additional specialist organisations and institutions for the blind and partially sighted, responsible for a total of 10,000 members, are DBSV corporate members.

The work of DBSV is grounded in the understanding of equal rights for and equal participation of blind and partially sighted people as a human right. Therefore, DBSV works towards the full implementations of the United Nations Convention on the Rights of Persons with Disabilities in Germany and in Europe.

Role in the project:
The DBSV is the vulnerable group partner within the SHIFT Project and as such serves to ensure accessibility and inclusion through insider experience of those directly affected. Although nominally representing the blind and visually impaired, we see it as our mission to lend a voice to all vulnerable groups standing to benefit from this effort to transform cultural heritage into something that everyone can experience and enjoy without barriers.

DBSV DEUTSCHER BLINDEN- UND SEHBEHINDERTENVERBAND EV

Germany

FORTH IDRYMA TECHNOLOGIAS KAI EREVNAS

Greece

The Human-Computer Interaction Laboratory (HCI Lab) of the Institute of Computer Science of FORTH is an internationally recognised centre of excellence in user-centred design methodologies, user interface software technologies and tools, with applications across the spectrum of societal domains and intelligent technological environments. HCI Lab’s mission is to actively contribute to an inclusive society, by carrying out pioneering research activities focused on creating frameworks, methods, tools and techniques that support the design, development, evaluation and deployment of (innovative user interfaces for) interactive applications and services that are accessible, natural, intuitive, and friendly to all users in the Information Society, including elderly people and people with disabilities. The R&D activities of the HCI Laboratory are Human Centred and are rooted in the principles of Universal Access and Design for All, encompassing a wide variety of application domains.

The Human-Computer Interaction (HCI) Laboratory of the Institute of Computer Science of FORTH leads WP3 “Audio engineering framework and multi-modal stimuli transformation”. Furthermore, it will be responsible for the technical development of SHIFT technologies, and in particular haptic techniques for 3D digital asset perception as well as an accessible framework of inclusive museum exhibits, acting as key enablers for digital inclusivity of cultural heritage for different target groups of the project, such as people with visual impairments.

HERITAGE INITIATIVE FOR HERITAGE CONSERVATION

Greece

The Heritage Management Organization (HERITΛGE) was established in 2008 to enable key heritage managersto independently transform heritage assets from decaying objects of study to dynamic sources of learning, community identity and economic development through targeted training.

We train professionals in the management of heritage sites, independently of project specifics. Training practitioners in the essential skills and best practices which define heritage management is at the heart of the HERITΛGE mission.

Since its founding HERITΛGE has trained hundreds of individuals and organizations in more than 90 countries and is now on course to impact a quarter of global heritage hotspots by the year 2025.

Role in the project:
HERITΛGE will lead the roadmap for building the SHIFT project business strategy and market validation for post COVID-19 recovery. The organization will contribute economic models for monetising cultural heritage supported by end-users.

HERITAGE will also contribute to introducing SHIFT and demonstrating its pilot use cases, as well as to the validation of the platform for museums, libraries, and content creators.

MDS MASSIVE DYNAMIC SWEDEN AB

Sweden

Massive Dynamic Sweden AB is a Swedish creative SME that innovates in research and development. The company was established and operates in Stockholm, since 2020 with the aim to develop research-oriented solutions, software, and methodologies. MDS has established collaborations and active contracts with partners in projects involving disaster management, computer vision as well e-learning and scenario-based training for experts and professionals in different fields.

MDS is leading the efforts in Deep-learning architecture and more specifically the design and implementation for foreground/background object detection. This task aims to develop an algorithm to distinguish between foreground and background objects from pictorial cultural archives. Addressing the task as an instance-aware semantic segmentation, the task will develop unsupervised method that will be capable of generating high-quality saliency maps to give a coarse distinction between the foreground and background objects of the image, using advanced RNN and RBN based architectures. Moreover, MDS is participating in Physics informed deep-learning network architecture in collaboration with Queen Mary University London aiming to generate motion-sequences for foreground objects using GANs and action sequence recognition within CH video repository.

QMUL QUEEN MARY UNIVERSITY OF LONDON

United Kingdom

Queen Mary University of London (QMUL) is one of the UK’s leading research-focussed higher education institutions. Recently, QM have invested £98M in new facilities and infrastructure over the past five years to offer students an exceptional learning environment QMUL has strong links to the industry and a substantial track record on supporting entrepreneurship. QMUL hosts one of the UK leading research groups in multimedia signal processing and computer vision – the Multimedia and Vision (MMV) research group. The MMV was formed in September 2000 to conduct high-impact research in image, video processing and computer vision. The vision of MMV was to enable cognitive interpretation and understanding of multimedia systems technology and to spread excellence through teaching and outreach. The group has participated and coordinated several EU funded projects including RACE, MACT, ACTS MOMUSYS, PANORAMA and Custom TV; Esprit, SCHEMA, BUSMAN, NoE K-Space and the COST292 Action. It was one of the main contributors and steering member of FP6 IST Integrated Projects aceMedia and MESH, co-ordinator of the STReP EASAIER and a partner in RUSHES. In FP7 ICT, the group was a core partner in the STRePs Papyrus and APIDIS, the NoE PetaMedia, REVERIE (technical coordinator), NoE 3DLife (Coordinator), SARACEN, NextMedia and MISSA. Similarly, the group has also participated in several key research projects in security domain including FP7 SEC NoE VideoSense, FP7 ADVISE and FP7 LASIE. Overall, the group has published over 500 journal papers, most of them in IEEE and IEE Transactions in the field over 500 refereed conference papers and secured over £10 Million grants funding from various sources.

Role in the project:

QMUL contributes to SHIFT project in leading the research activities on “Tools for authoring augmented hypermedia and gesture recognition for enriched accessibility” (WP2) in collaboration with MDS, SIMAVI and FORTH. Additional activities of QMUL includes contribution to the drafting of user requirements (WP1), development of digital rights management (DRM) toolkits for the management of digital assets generated and shared within the SHIFT project.

SMB STIFTUNG PREUSSISCHER KULTURBESITZ

Germany

The National Museums in Berlin (Staatliche Museen zu Berlin – Preußischer Kulturbesitz) almost housed in historical buildings, constitute a Universal Museum for the preservation, research and conveyance of treasures of art and culture of the entire history of humanity. Its collections, shared over 15 different museums and located all over the City of Berlin, embrace European and Extra-European Art, Archaeology and Ethnology, ranging from the first steps of mankind to contemporary fine arts, from Nefertiti in the Egyptian Museum to Josef Beuys in the National Gallery. Many of these Collections are ranking high among the world’s most important museums. Museum Island has been awarded World Cultural Heritage by UNESCO in 1999.

Operating under the legal umbrella of The Prussian Cultural Heritage Foundation (Stiftung Preußischer Kulturbesitz), SMB-PK is financed and counselled by the federal German government and all of the federal states collectively. It is regarded as a national institution of cultural federalism in Germany.

The executive board comprises different departments which are charged with the co-ordination of centralized services, conveyance and research in the museums as well as with public outreach. With its different departments and museums SMB-PK shall act as partner in the SHIFT project, always in close coordination with special collections, such as the Museum of Photography, the Gallery of Old Masters or the Pergamon Museum.

The task of SMB-PK in SHIFT concerns the user studies involving their lifecycle from requirement analysis until validation. SMB-PK will also be actively involved in setting up use-case scenarios, exhibits and the museum case study.

SMB-PK will promote and disseminate SHIFT concepts in conferences, workshops and online visitors approach. It aims to acquaint the public audiance as well as the international colleagues and professional consultants with the attractive potential of the SHIFT technologies.

SOMKL MAGYAR NEMZETI MUZEUM

Hungary

The MNM Semmelweis Orvostörténeti Múzeum (Semmelweis Medical Museum of the Hungarian National Museum) collects, preserves, and exhibits tools, documents and artefacts related to the history of disease, medicine and body.

Our overall topic concerns about any familiarly typical national narratives, in contrast it maintains and provides a new, unusual, almost unknown, yet universal focus of the history of Europe and the world.

Role in the project:

A cultural heritage institution, a museum, a leading end-user with ownership of CH content, contributing to the requirements for digital tools developed in SHIFT and participating in the promotion the project outcomes, also WP1 leader.

Our objective is to introduce not only state-of-the-art technologies into the museum world (which could be done by typically sheer routine), but to develop methods, set up networks and bubble up hubs for long running cooperations to increase our reach towards the public, for our public engagement to create practices of a better understanding of the best innovative methods available, and to make them applicable for creating new experiences for both museum staff, experts and the general public.

Technical University of Munich (TUM)

Germany

The Technical University of Munich (TUM) is one of Germany’s top
universities. It has unique profile with natural sciences, engineering, life sciences, and medicine at its core. TUM is committed to addressing the major challenges facing society in the 21st century in areas such as energy, climate, and environment, natural resources, health and nutrition, communication and information, mobility, and infrastructure. Klinikum rechts der Isar (TUM-MED) is the TUM university hospital. Founded in 1834, TUM-MED today stands for state-of-the-art medicine, research, and advancement. A key research group within the TUM-MED is the Chair of Health Informatics (CHI), which focuses on the development and application of artificial intelligence (AI) techniques, particularly deep learning, to address critical challenges in healthcare and wellbeing. This group’s research endeavors have a significant impact on AI-driven solutions for disease diagnosis, treatment, rehabilitation, and health monitoring. Areas of expertise include affective computing, human activity recognition, and physiological signal processing. By engaging in interdisciplinary research, the CHI demonstrates a strong commitment to advancing the AI field and developing innovative solutions with real-world impact.

Role in the project:
TUM-MED plays a prominent role in SHIFT, leading and contributing to multiple tasks that involve natural language processing (NLP), deep learning algorithms, and multimedia content analysis. As a leader in several tasks, TUM-MED will develop comprehensive textual representations of cultural heritage (CH) assets. Leveraging its expertise in NLP and deep learning, TUM-MED will focus on advancing state-of-the-art techniques to deliver impactful outcomes in these areas. Moreover, TUM-MED will lead the development of multimedia content analysis algorithms, suitable for processing cultural archives and enabling content enrichment, semantic segmentation, and storytelling. By investigating and extending existing deep learning architectures, TUM-MED aims to maximize salient information extraction from mainly textual modalities. TUM-MED will also contribute on assisting the development of interfaces capable of integrating with existing cultural asset repositories, content management systems, and digital rights management systems.

Work packages.

WP1

SHIFT user requirements analysis for Cultural Heritage accessibility and inclusion

Objectives

To study and formalise the stakeholder requirements on cultural curation, accessibility, inclusion and storytelling
To identify the IT infrastructure commonly available among stakeholders to support SHIFT component installation
To develop usability evaluation guidelines and acceptability metrics
To document content distribution processes from end-user organisations to technology providers
To gather, curate, edit, publish UGC (or professional) contents at any scale

WP2

Tools for authoring augmented hypermedia and gesture recognition for enriched accessibility

Objectives

To design and implement novel deep-learning architectures for foreground and background segmentation from CH archives

To investigate physics-informed constraints in training deep-learning algorithms for foreground object motion sequence generation

To develop automated motion sequence generator for foreground object using GANs
To recognition actions from video sequence(s) for generating audio captioning

WP3

Audio engineering framework and multi-modal stimuli transformation

Objectives

To devise comprehensive textual representation algorithms based on novel NLP approaches of cultural heritage assets based on big volumes of descriptions and semantics

To develop a novel text/video to speech tool able to convey the visual / textual stimuli of the source to emotional augmented audio streams

To model temporal evolution of language resources for enriched cultural content curation
To implement haptic interaction with 3D digital tangible and intangible cultural heritage assets, augmenting the Users

Experience (UX) with new interaction paradigms that can be used in situ or remotely

To provide a comprehensive intuitive and accessible for all multimodal storytelling of cultural heritage assets

WP4

Cross-modality curation of cultural content using inclusion by design

To define a set of content pre-processing features and implement the toolbox for metadata extraction
To develop novel approaches for multimedia curation based on association by design
To implement standardised APIs for stakeholder system integration with SHIFT repository
To develop metadata interoperability formats based on association by design for long-term preservation of cultural assets

WP5

SHIFT architecture design, integration, validation, and demonstration

Objectives

To design and develop a valid architectural system and implement a functional platform and define integration patterns
To ensure full functional integration based on the architectural design
To implement a back-end system that currently supports “content management services” and user management for all actors
To perform platform and tools wide validation with specific end-user pilots

WP6

Dissemination and communication

Discover whether the system is right for the market and find bottlenecks and possible oversights. Ensure extensive adoption by stakeholders and the public by disseminating results and outcomes in an effective manner. To develop a sustainable business for the platform, the multimedia video clip and storytelling products

Grow our contributor communities and welcome new audiences to join our brand storytelling, by encouraging them to share their experiences though photos, videos and live video feeds
Deliver authentic, immersive, and relevant content through interactive galleries, social activities, promotional contests and more

WP7

Project coordination

The project management encompasses contractual issues, technical, administrative, finance, communication and knowledge management inside the project, and external relationships between the project and the EC. The Project Coordinator SIMAVI will lead the overall project management including the system integration and technical deployment in the cloud, but it will be assisted by the members of the Steering Committee and the WP leaders. The project coordination aims to guarantee a timely implementation of the respective activities and achievements of the project objectives and the quality of the project deliverables. The objective of this WP is to ensure that the project achieves its goal within the budget and scheduled deadlines.