Automating Compliance and Security for the Department of Agriculture, Food and the Marine with Smart Text

DAFM handles applications and payments for farming funding grants. DAFM receives 30,000-40,000 grant application documents annually, which often include sensitive information. Farmers often failed to classify these documents as sensitive, and agents sometimes uploaded documents to the wrong application. As a result, sensitive personal data was exposed to BPS employees or even other farmers, creating compliance risks.

DAFM’s Basic Payments Scheme (BPS) handles applications and payments for farming funding grants. It is responsible for issuing €1.2 billion yearly to 120,000 farmers, in line with the European Union (EU)’s General Data Protection Regulation (GDPR) that protects the personal data and privacy of EU citizens. Each year, BPS receives 30,000-40,000 grant application documents submitted to its website by farmers or their agents. These documents often include Personally Sensitive Information (PSI), such as birth certificates, legal documentation, or medical information. However, farmers often failed to correctly classify documents as containing PSI. Agents—some of whom serve dozens of farmers—sometimes uploaded documents to the wrong application. As a result, highly sensitive personal data was exposed to BPS employees or even other farmers.

To mitigate the risk of financial penalties and reputational damage from major GDPR breaches, DAFM sought to replace its legacy data system and processes with an intelligent solution that would help correctly identify PSI and automate the detection of breaches.

BPS worked with Version 1 to outline requirements for an intelligent, automated approach to grant application processing. After evaluating several vendors, DAFM and Version 1 decided to use Red Hat OpenShift, supported by Red Hat 3scale API Management and Red Hat’s single sign-on (SSO) technology.

Smart Text was created, an ML text analytics solution that can extract metadata from submitted documents to correctly identify PSI, in just a few weeks. To meet EU requirements for data ownership, this solution runs on Red Hat OpenShift on-premise in DAFM’s datacentre. 3scale API Management, part of Red Hat Integration, uses application programming interfaces (APIs) to connect the platform with DAFM’s citizen-facing application. Red Hat’s SSO technology supports protected mobile access for applicants using OpenID Connect and OAuth authentication. Its success using open source technology to create the Smart Text solution led to DAFM’s recognition with a 2021 Red Hat Innovation Award.

Smart Text, hosted on Red Hat OpenShift, uses real-time artificial intelligence (AI) and ML capabilities to extract metadata and other contextual information from unstructured grant application documents. These documents, including scans of handwritten or typed letters, are analyzed for sentiment, semantically similar words, topics, and entity names.

A DevOps work approach has also helped DAFM significantly reduce manual administration and management. Collaborating on a single platform, teams have better visibility into project statuses—even while working remotely. 3scale API Management lets DAFM share and reuse services and applications as APIs among teams through a user-friendly interface. Each component API is deployed as a container with open source libraries included for simple reuse and redeployment.

When Smart Text identifies a potential data privacy breach, the document is isolated and only authorised individuals can view the contents and redirect the document. As a result, DAFM can drastically reduce the number of breaches—and the associated reputational damage and financial penalties. Additionally, DAFM reduced development time from weeks to days with automation, APIs, and DevOps and improved security and stability with enterprise technology and support.