| | | | Bi-Annual Newsletter: Fifth Edition
| | About ROXANNE
Fighting organized crime remains an important but challenging task for Law Enforcement Agencies (LEAs) in all countries. The ROXANNE advanced data analysis solution, developed using a Privacy and Ethics-by-Design approach, aims to enhance and assist police investigations, while reducing the cost and burden on the society owing to organized criminal activities. In the ROXANNE consortium, LEAs, industry and academia, are joining forces in the fight against organized crime.
The main goal of this project is to develop an interactive platform with AI that will combine advanced text, speech, and language technologies, along with network analysis all into one platform. The ROXANNE project fully complies with the European legal framework and ensures that both privacy and individual rights are duly respected. ROXANNE aims not only to improve and assist police investigations but also to speed up investigative processes and support LEA decision-making. | | | |
Romania’s Ministry of Internal Affairs & ROXANNE Project
When joining the ROXANNE project, Romania’s Ministry of Internal Affairs sought out new and innovative ways to fight criminal organized groups. As criminal groups evolve, so must we. Perpetrators strive to find new ways to conduct their operations, they learn, and they adapt to the newest trends and opportunities. In order to be able to compete with them, we must do the same. In these last years, the use of AI (Artificial Intelligence) by law enforcement authorities has relied on the premise that AI in the criminal law field would reduce certain types of crime and lead to more objective decisions. That is where ROXANNE project comes in play, as it gathers academia, industry, and law enforcement agencies under the same roof and allows them to share knowledge and new technologies. The ROXANNE project has capitalised a lot of aspects of AI technologies.
As innovative technologies come to light, it is our duty to steer their development for the good, and to do so we must share first-hand knowledge of criminal activities. The COVID-19 pandemic hindered our daily tasks but created more opportunities for perpetrators. With a war near our borders, new challenges emerged. The traditional way to solve the issues is slow and requires a great number of resources. Two years into the ROXANNE project, we feel like our requests have been met and our tasks were carried out responsibly. Seeing the 2nd field test hosted by NFI and being part of the data collection workshop hosted by ZiTiS made us understand the technical limitations and the mindset of the developers.
We strongly believe that our expertise will contribute to the final outcome, an advanced technical platform which uses new tools to uncover and track organized criminal networks and which would reduce the investigation time and resource consumption. We are eager to continue testing the platform and provide the necessary feedback in order to reach the goal of the project. | | | |
Consortium Updates During the last 6 months ROXANNE consortium organized two technical meetings in a hybrid mode. These meetings were hosted by HENSOLDT Analytics in Munich, Germany and by IDIAP in Martigny, Switzerland, respectively.
The ROXANNE Consortium Meeting on 2 Nov 2021 was attended by 56 participants from 17 countries, including the representatives of LEAs, SMEs, academic and industrial partners. Partners were presented with the Autocrime ultra-light platform, discussed all work packages updates and plans for the last year of the project. There was also a discussion on the second field test. The event was followed by a 2-day workshop, to collect additional data for the ROXANNE Simulated Dataset. This workshop provided an opportunity to discuss and define the future direction of the ROXANNE project with LEA partners. | | | | Figure 1: ROXANNE Consortium Meeting Participants
| | The ROXANNE Technical Meeting held on 22-23 February 2022 was focused on the project's technical aspects as it is moving towards its final stage. For those, who could not join in person due to travel bans, a short on-line session was dedicated. | | | | Figure 2: ROXANNE Technical Meeting Participants | | At the beginning of December 2021, ROXANNE representatives also participated in ISS World Europe, the world's largest gathering of Regional Law Enforcement, Intelligence and Homeland Security Analysts, Telecoms as well as Financial Crime Investigators responsible for Cyber Crime Investigation, Electronic Surveillance and Intelligence. ROXANNE Simulated Dataset (ROXSD) was presented during the last day of ISS symposium, by HENSOLDT Analytics and PHONEXIA representatives: Gerhard Backfried (HENS) and Květoslav Malý (PHO). Our colleagues introduced ROXSD, which was prepared by the ROXANNE project with assistance from several LEAs, to facilitate the development and evaluation of novel tools and technologies for criminal investigations. | | | | Figure 3: ROXANNE representatives at ISS World Europe | | | |
Progress in the last 6 months Research work during the past 6 months We continued the integration of more technologies and components into our research platform, Autocrime, a tool to process audio conversations from phone calls and an interface for the interaction between speech recognition, natural language processing, and network analysis components. Autocrime is a lightweight platform that benefits from state-of-the-art technologies, runs on a CPU, and can be efficiently used on various datasets, including our crime simulation dataset, ROXSD. Currently, our LEA partners use and test the platform, and the goal is to publicise it at the end of the project. Besides Autocrime, the technical partners improve the existing technologies and develop new ones. For example: - Improve speaker identification (SID) accuracy using more advanced and state-of-the-art methods;
- Improve the voice activity detection (VAD) method;
- Boost automatic speech recognition (ASR) by using state-of-the-art deep neural network methods;
- Develop text-dependent speaker identification and role recognition technologies based on the output of ASR technologies and adapt them to the ROXANNE domain;
- Consider prior information in speech recognition, especially in the form of graph-structured data, and as feedback from network analysis technologies. This line of research has been recently started as a collaboration between speech recognition and network analysis partners;
- Improve named entity recognition (NER) and co-reference resolution methods on the output of automatic speech recognition;
- Extract relations based on a list of more essential ones from the LEA perspective;
- Extract mentions from the call contents, expand the call network to contain those mentions, and evaluate network analysis technologies such as link prediction;
- Develop an inductive link prediction method in network analysis based on attributed node embedding and deep neural networks;
- Develop a deep random walk framework for node matching in cross-network analysis and multi-modal networks;
- Develop an unsupervised anomaly detection in the nodes of the network data;
- Initial discussion and preparation to consider geolocation information to improve several of the technologies mentioned above like NER or link prediction.
We have systematically assessed many of our developed technologies on ROXSD and continue the assessment with the latest version of simulated data and the development of new technologies. In addition to ROXSD, we work on two anonymised real cases provided by an LEA partner to evaluate network analysis tools better. One dataset is a phone tap case, and another is a burglary network.
Multimodal document indexing and retrieval In the first phase of the project, ROXANNE partners developed and combined several audio, text and network analysis algorithms to support LEAs in their understanding of criminal networks. In particular, voiceprint extraction and clustering, automatic speech recognition, named entity recognition and various network analysis techniques were applied to the ROXSD dataset consisting of hundreds of calls between target and non-target characters involved in a drug smuggling and drug production scenario to demonstrate the added value of these techniques in accelerating investigator’s work thanks to the automated structuration, semantization, and efficient visualization of this large set of initially unstructured data. To go even further, the Autocrime platform is currently being enriched with additional processing capabilities enabling to process additional document types such as images or videos coming either from seized phones or computers or retrieved from some DarkNet forum known for hosting illegal activities and related to the case. In this scenario, the number of videos and images may be very large, most of them being not relevant to the case, and finding the informative image or video is very time consuming and prone to non detections due to the inevitable decrease of user’s attention after a few hours of viewing.
To support LEAs in accelerating their identification of relevant documents, the Autocrime platform is being enriched with a multimodal indexing capability aimed at providing a multimodal document similarity evaluation engine relying on a set of signatures (or embeddings) representative of the document’s content with respect to the various considered modalities namely voices, faces, scenes and named entities. Each video processed by the Autocrime platform is split between its audio and image streams. The audio stream gets processed through speaker diarization (the process of clustering and temporally segmenting the voices heard in the video), automatic speech recognition, named entities recognition (mentioned names, locations, dates) while the image stream leverages face detection and embedding extraction and global scene embedding extraction to enable scene similarity matching (ability to match two pictures sharing some common background). Thanks to the audio processing of the videos, the videos involving characters whose voice cluster is already present in the ROXSD network will be automatically associated to the corresponding nodes leading to additional links between voice clusters heard in the same video, and additional nodes of characters heard in the videos but not heard in the ROXSD calls. The visual similarity evaluation can then be used to suggest additional edges between nodes attached to images or videos involving similar faces or scenes. Corresponding matches are either validated or rejected by the operator thus confirming the suggested link or not. Finally, the operator may select part of the totality of modalities attached to a node (e.g., voice, face, scene, named entities) and conduct a multi-modal document search on indexed external documents (e.g., videos, calls, images). Thanks to the ranked list of retrieved documents, his attention is focused on the documents which have the highest probability to share some common features with the initial query.
To conclude, with these additional capabilities, the Autocrime platform’s users will be able to process a larger diversity of document types, will be supported in the identification of additional relations between persons of interest thanks to visual matches, and will more rapidly and with an expected higher detection rate find the “needle” document among a haystack of non relevant documents thus accelerating investigations and increasing their successful resolution rate.
ROXSD: The latest developments in our Simulated Dataset Investigations of organized crime cases are resource and labour intensive, since telephone conversations intercepted between wire-tapped suspects need to be appropriately processed, transcribed, potentially translated, and finally mapped into a knowledge graph. The legal framework is also important for regulating how law enforcement agencies can process personal data. In an effort to foster and boost the work on the support for automation of such investigations and to allow further contributions, we have created ROXSD, a simulated dataset representing a fictionalised organized crime case in a realistic manner. It aims to meet the constraints of a real scenario and to realistically represent it at an appropriate level of complexity, while also remaining freely accessible for research. Due to its rich set of data and associated meta-data, ROXSD lends itself to a multitude of applications and can be employed for a variety of evaluation (and development) purposes. This concerns a series of technologies from the field of NLP as well as from the field of network analysis. Regarding the NLP technologies, both audio, and text-based aspects, can be addressed. Regarding network analysis, the previous technologies can be used to establish/hypothesize relations (and their evolution) between actors, locations, and conversational contents (like the mention of locations for handing over of substances, etc.). These are the used technologies: - Speech Processing: Speaker Clustering and Identification, Age and Gender Identification, Language and Dialect identification, Automatic Speech Recognition, Keyword-Spotting
- NLP: Named Entity Recognition (location, organizations, etc.), Topic Detection, Identification of unknown 2nd party in calls, Detection of mentions of 3rd parties
- Network Analysis: Network comparison metrics, Knowledge graph representation, Dynamic link prediction
The scenario involves the Prague anti-drug unit of the Czech police investigating three hypothetical cases at the same time: a first drug distribution case involving Czech and Russian students, named DDA (Drug Distribution A), a drug lab, ran by Vietnamese suspects, named DLA (Drug Lab A), and another drug distribution case which involves individuals speaking German, named DDB (Drug Distribution B). The ROXSD data set is recorded in several stages. So far, two data collection sessions have taken place and a third is in preparation, afterwards the simulated data collection will be completed.
The first collection was technically managed by the company Phonexia, which prepared an application for recording called Phonexia Twilio Application. With the cooperation of partners, over 100 stereo recordings were recorded according to the fictional story and metadata were prepared based on the initial requirements of the technology partners. The second data collection was managed by the Central Office for Information Technology in the Security Sector (ZITIS), who are an LEA research organisation. Thus, it was possible to perform recordings on actual interception equipment, which allowed developers to test the functionality of their technologies with real and not laboratory quality recordings. The audio recording took place directly at the ZITIS premises in Munich at the end of 2021. At present, the obtained recordings and metadata are still being processed and prepared for further use. Currently, the consortium focuses mainly on extracting the content of recordings for the purposes of automatic transcripts, network analysis, mention network and others. The number of audio recordings after the second recording is still insufficient and therefore further data collection will take place in the ZITIS premises. For the third data collection the consortium plans to extend the recording period and is currently looking for volunteers to take part in the event. The planned date of the upload is May 2022. Immediately after the upload, the data will be processed so that it is available to the user as soon as possible for technology evaluation and testing.
ROXSD will be available to researchers and LEAs for research and non-commercial use with a signed agreement and we plan to make ROXSD available as a resource via the European Language Grid.
ROXANNE towards recognizing names entity and topics The organized crime investigation involves telephonic conversations intercepted between tapped suspects. We have curated an associated meta-data in ROXANNE, which we leverage for many NLP applications, namely Names Entity Recognition (location, organizations, etc.), Topic Detection, Identification of unknown 2nd party in calls, Detection of mentions of 3rd parties. We have developed a NER module with the FLAIR model, which exhibits a 92% F1 score on our target data. The module can recognize persons and locations in the transcribed and translated calls. We also develop a topic detection module that classifies a call in [Drugs', 'Work Conversations,' 'Other,' 'Family Conversations,' 'Meeting,' 'Money'] using zero-shot learning with 60% accuracy. In the future, we are leveraging our experiments on automatically generated transcripts obtained by speech-to-text module to detect the named entity and topics in real time.
| | | |
Ethics in ROXANNE
Ethics-by-design approach in ROXANNE The aim of Ethics by Design is to incorporate ethical principles into the development process so that ethics are embedded into the technology design process, meaning that ethical challenges are addressed as early as possible and followed up closely during research activities. It is intended to mitigate ethical issues before they arise, rather than trying to fix them later in the process.[1]
In the case of ROXANNE, this means that the research project is carried out and the resultant platform is built in such a way that respects privacy, ethical standards, and legal constraints. Such an approach allows many of the issues raised by LEA data analysis to be mitigated before law enforcement use the ROXANNE platform, meaning that ROXANNE partners have considered several possible scenarios to reduce the risks and impacts of illegitimate uses of the platform. To ensure a successful Ethics-by-Design approach in ROXANNE, the consortium has support from both an Internal and External Ethics Board.
The Internal Ethics Board is comprised of experts from project partners who meet to discuss and decide upon ethical issues arising in the project and provide oversight to the in-depth ethics work carried out in the project. The External Ethics Board includes experts in ethics, law, and technology, who provide advice on ethical issues arising from the project and proposed by project partners. Partners can bring issues to the Ethics Boards for discussion as the research develops, so as to enhance their own research with the expertise of the Ethics Boards.[2]
[1] Ethics By Design and Ethics of Use Approaches for Artificial Intelligence [2] Ethically developed technologies for safer societies – The ROXANNE project case study
Internal and External Ethics Board’s updates Meetings of both Ethics Boards cover an array of different issues. A key area for discussion at recent Internal Ethics Board meetings has been about ensuring that research into tools for analysing video data including faces is conducted ethically, respects the privacy of any data-subjects, and minimizes risks of misuse and mass surveillance if the tool ever reaches maturity. This has also been discussed with the External Ethics Board. Other topics discussed with the External Ethics Board include mitigating issues of processing sensitive data, minimizing risks of misuse, and ethical management in the ROXANNE project. Future discussions are expected to cover how ROXANNE has approached dealing with bias in technologies, and minimizing risks associated with providing tools to potential future customers.
Privacy-by-Design in ROXANNE technologies Privacy is an important ethical, social, and legal issue. Yet, some infringement on a person’s privacy by law enforcement agencies (LEAs) might need to take place for effective criminal investigations; this is generally seen as socially acceptable and in compliance with human rights law when it is necessary, proportionate, and done according to legal rules. As such, there are limitations on LEAs processing data about people’s private lives that need to be respected, and the way we design processing technologies is part of that respect.
Read more here.
Towards a trustworthy ROXANNE platform Funded under the H2020 programme supporting ground-breaking research and advancing European excellence, ROXANNE aims to support law enforcement authorities (LEAs) fight crime and terrorism by facilitating the analysis of criminal data. To this end, the ROXANNE platform combines innovative data analysis capabilities, including speech and language technologies, visual analysis and network analysis, to help identify perpetrators. ROXANNE’s innovation lies in the bi-directional interaction between the multimodal technological processes integrated in the platform.
The analysis process further benefits from prior knowledge available to investigators for increased accuracy of results that contribute to advancing the case. With Artificial Intelligence (Al) at the core of the ROXANNE platform through the underlying algorithmic models, the project team adopted ethics and privacy by design approach in its research work in order to develop an ethically, legally and socially sound final result.
Read more here.
| | | |
Diverse aspects of ROXANNE
ROXANNE in collaboration with sister projects ROXANNE has been involved in monthly meetings with ethical and legal experts from the FORMOBILE and LOCARD projects. Colleagues have discussed ethical and legal challenges they have been facing in projects and shared advice on how to deal with shared issues. Continuing the fruitful discussions, partners from ROXANNE, FORMOBILE, and LOCARD have been collaborating on a joint paper analysing the current drafts of the proposed eEvidence regulation at EU-level. More information on this is provided below. The findings will also be presented at the FORMOBILE final conference 28-29 April 2022, with the ROXANNE contribution presented by Dr. Joshua Hughes from Trilateral Research.
FORMOBILE project updates FORMOBILE Project’s Final Conference will be organised in a hybrid model on 28-29 April 2022. ROXANNE project will attend this event through TRI and NFI partners. FORMOBILE will showcase the final results of the 3-year initiative at this conference. By providing detailed insight into the outcomes of the project and future opportunities to utilise the results of the FORMOBILE initiative: https://formobile-project.eu/news
LOCARD project updates Currently, LOCARD is finalising the testing and validation phase (month 35) and the demonstration to external stakeholders is planned to start soon. The dissemination tasks in conferences and webinars, publication of research articles and progress with stakeholders and sister projects are ongoing. In particular, the joint paper written by LOCARD, ROXANNE & FORMOBILE will be presented at the CEPOL Conference next June. Also, LOCARD has received the official notification from ISO to be a liaison organisation. Moreover, LOCARD training sessions are available for stakeholders: https://tutorials.locard.motivian.com/ and access will be also provided from the project’s website. More information can be found here: https://locard.eu/news-events/news or through email: asolanas@apwg.eu or agusti.solanas@urv.cat
CC-Driver Policy Toolkit Wiki Workshop The CC-Driver project seeks to understand the drivers of cybercrime and research methods to prevent, investigate and mitigate cybercriminal behaviour.
CC-Driver is developing a policy tool wiki, including a set of good cybersecurity policy practices, with a special focus on young people, and a cybersecurity policy framework outlining the different types of cybercrime, including cybercrime-as-a-service, and the extent to which particular bits of legislation or regulation address those different types of crimes. This wiki will be presented for the first time to an audience of law enforcement authorities and policymakers in an online workshop on 9 May, 10-12 CEST.
Please register here: eventbrite.de/e/cc-driver-policy-tool-wiki-workshop-tickets-305670126577
| | | | | |
ROXANNE at CEPOL conference Our proposal to CEPOL online conference (as a joint effort between LOCARD, ROXANNE and FORMOBILE projects) has been accepted. The paper: “e-Evidence: Collection, Analysis, and Sharing: An evidence-based policy perspective by the EU funded research projects LOCARD, ROXANNE & FORMOBILE” will focus on the following areas: - Current legal framework regulating collection of eEvidence data (Law Enforcement Directive, rights to privacy and data protection),
- How the proposed eEvidence regulation will change the status quo,
- Analysis of whether the proposed eEvidence regulation is in conformity with the current legal framework.
The conference will take place in June 2022 at Mykolas Romeris University, Vilnius, Lithuania (https://conference-digital.cepol.europa.eu/cepol-research-science-conference-2021-mru-vilnius/). ROXANNE at EAFS 2022 ROXANNE will participate on EAFS2022 conference. EAFS (European Academy of Forensic Science) conference will be organised this year in Stockholm, Sweden on May 30th–June 3rd, 2022. The conference is considered as one of the largest European events focusing on forensic science. EAFS in 2022 is organised by the National Forensic Centre (NFC), a department within the Swedish Police Authority, under patronage of the European Network of Forensic Science Institutes (ENFSI). ROXANNE will be presented by AEGIS, Idiap and NFI at this conference, presenting results of the project through the demonstration as well as oral presentation. The abstract of our presentation is given below:
“Law enforcement practitioners need to be able to efficiently and successfully investigate a large number of criminal cases at the same time, while following relevant ethical guidelines. This calls for an efficient way to analyse potentially relevant information and automate the most time-consuming tasks. The ROXANNE autocrime platform will enhance criminal network analysis capabilities by providing a) a suite of multi-modal technologies for extracting evidence and actionable intelligence and b) an effective and intuitive forensics visualisation scheme that streamlines the discovery, presentation and evaluation of these insights. Key backend technologies of autocrime include Speaker Identification, Automatic Speech Recognition and Natural Language Processing, as well as criminal Network Analysis for identifying communities, etc. Furthermore, autocrime offers an intuitive user interface supporting (criminal) network analysis for understanding how entities interact and uncovering patterns of interactions on the temporal dimension, equipped with advanced filtering for narrowing down the results according to users’ criteria. Taking for granted that human expertise and judgment is key, autocrime allows practitioners to judge and update the outputs (e.g., by suggesting changes to the speaker identification/ clustering) so that the AI-based technologies are improved in the long run.The added value of autocrime platform has been evaluated using a synthetic dataset for a fictional drug-dealing case that includes more than 100 target phone calls and related metadata (i.e., CDR info) and is currently being extended to process additional document types such as images or videos.”
Please visit: https://www.eafs2022.eu/
| | | | | | | | | | | | | | | |
