/ Development  

Unlock your content with Apache Tika; Extract document metadata from OPA

Hi there “Process Automation” fans,

Welcome to a new installment of “Process Automation” tips.

I’ve been in the content space now for almost 20 years and have seen a lot of tools passing by that have a relation with content. Mostly big corporate tools like the tools from OpenText; however, my eye fell (during a browse session for a different R&D session) on a fascinating project by the name of Apache Tika. I never heard of it during my complete career. Until now! Is it too late? I don’t know…I see this product has a similar nearly 20-year history! So, I’m just curious as Tika™ can detect and extract metadata/text from over a thousand different file types (such as PPT, XLS, and PDF) which makes it a powerful tool to turn your content into AI-readable knowledge! Think in terms of:

  • MIME/type detection
  • Text extraction
  • Metadata extraction
  • Document structure extraction
  • Attachments / embedded documents
  • OCR integration
  • Normalization of many file formats

So, it’s worth spending some time with Apache Tika™…

Let’s get right into it…

What will be the use-case? Well, when a document enters the solution we want to trigger some business logic to extract its text/metadata! My first thought was the ‘Document Received’ construct in a lifecycle (read about it here). However, why not just add a ‘Rule’ building block on a relative ‘Contents’ entity for a ‘Case’ entity which triggers an ‘OnCreate’ event that starts a BPM…aha! This BPM can then read the content (in raw document bytes!), call Tika (over the HTTP connector!) and receive its magic into the properties of our document; properties like case_content_text and case_content_metadata.

I see one challenge here which is grabbing our content in “raw document bytes” from a BPM perspective! Why? Well, Tika Server’s endpoints accept an ‘InputStream’. This could also be a Base64 decoded string! So, don’t send JVBERi0x..., but send %PDF-1.7... OR save it on the server first as a file (which you also need to clean)!

Great insights and thoughts, but we first need to have an Apache Tika instance up and running!

Boot up your RHEL VM and install Apache Tika with these simple commands:

1
2
3
4
sudo mkdir /opt/tika
cd /opt/tika
sudo curl -O https://dlcdn.apache.org/tika/3.3.2/tika-app-3.3.2.jar
sudo curl -O https://dlcdn.apache.org/tika/3.3.2/tika-server-standard-3.3.2.jar

Yes, you need Java; those JAR files are self-executable!

Let’s do a first run as “standalone” (CLI) app and make sure you have a document uploaded on the server (~/contract.pdf or /home/sysadmin/contract.pdf) where you run these commands:

1
2
3
java -jar /opt/tika/tika-app-3.3.2.jar --version
java -jar /opt/tika/tika-app-3.3.2.jar ~/contract.pdf
java -jar /opt/tika/tika-app-3.3.2.jar -m ~/contract.pdf

NICEEEEE! 😍 This first call gives the version, the second extracts the content, and the last one grabs the metadata (-m)!

…

Time to level up as Tika also has a server component. Have a play with commands like this which will give you a similar result; only this time over HTTP/REST calls:

1
2
3
4
5
java -jar /opt/tika/tika-server-standard-3.3.2.jar --host 0.0.0.0 --port 9998
curl http://192.168.56.107:9998/tika
curl -T ~/document.pdf http://localhost:9998/tika
curl -T ~/document.pdf -H "Accept: application/json" http://localhost:9998/meta | jq
curl -T ~/document.pdf -H "Accept: application/json" http://localhost:9998/rmeta | jq

You see me adding this: --host 0.0.0.0 --port 9998. That’s because Tika (by default) only listens to localhost!

This is my metadata response JSON from the document.pdf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
  "pdf:unmappedUnicodeCharsPerPage": "0",
  "pdf:PDFVersion": "1.7",
  "pdf:docinfo:title": "contract.txt - Notepad",
  "pdf:hasXFA": "false",
  "access_permission:modify_annotations": "true",
  "X-TIKA:Parsed-By-Full-Set": [
    "org.apache.tika.parser.DefaultParser",
    "org.apache.tika.parser.pdf.PDFParser"
  ],
  "dc:creator": "Antal",
  "pdf:num3DAnnotations": "0",
  "dcterms:created": "2021-02-01T13:57:05Z",
  "language": "ca",
  "dcterms:modified": "2021-02-01T13:57:05Z",
  "dc:format": "application/pdf; version=1.7",
  "pdf:overallPercentageUnmappedUnicodeChars": "0.0",
  "access_permission:fill_in_form": "true",
  "pdf:docinfo:modified": "2021-02-01T13:57:05Z",
  "pdf:hasCollection": "false",
  "pdf:encrypted": "false",
  "dc:title": "contract.txt - Notepad",
  "pdf:containsNonEmbeddedFont": "false",
  "Content-Length": "51744",
  "pdf:hasMarkedContent": "false",
  "pdf:ocrPageCount": "0",
  "Content-Type": "application/pdf",
  "access_permission:can_print_faithful": "true",
  "pdf:docinfo:creator": "Antal",
  "pdf:producer": "Microsoft: Print To PDF",
  "pdf:totalUnmappedUnicodeChars": "0",
  "access_permission:extract_for_accessibility": "true",
  "access_permission:assemble_document": "true",
  "xmpTPg:NPages": "1",
  "pdf:hasXMP": "false",
  "pdf:charsPerPage": "3631",
  "access_permission:extract_content": "true",
  "access_permission:can_print": "true",
  "X-TIKA:Parsed-By": [
    "org.apache.tika.parser.DefaultParser",
    "org.apache.tika.parser.pdf.PDFParser"
  ],
  "access_permission:can_modify": "true",
  "pdf:docinfo:producer": "Microsoft: Print To PDF",
  "pdf:docinfo:created": "2021-02-01T13:57:05Z",
  "pdf:containsDamagedFont": "false"
}

Nice, but these calls require having a document located on the server! Can we do this differently? Yes, you can play with these calls:

1
2
3
curl -X PUT --data-binary @~/document.pdf -H "Content-Type: application/pdf" http://192.168.56.107:9998/tika
curl -X PUT --data-binary %PDF-1.7... -H "Content-Type: application/pdf" http://192.168.56.107:9998/tika
some-command-producing-pdf-bytes | curl -X PUT --data-binary @- -H "Content-Type: application/pdf" http://192.168.56.107:9998/tika

--data-binary %PDF-1.7... is just an example; Don’t use it for a real PDF, which is not reliable; PDF bytes can contain binary values, NUL bytes, quotes, shell-sensitive characters, etc.

If you need more logging/config on the Tika-side: sudo vi /opt/tika/tika-config.xml

1
2
3
4
5
6
7
8
9
10
11
<?xml version="1.0" encoding="UTF-8"?>
<properties>
    <server>
        <params>
            <host>0.0.0.0</host>
            <port>9998</port>
            <logLevel>debug</logLevel>
            <returnStackTrace>true</returnStackTrace>
        </params>
    </server>
</properties>

You can now start Tika with this config XML like this: java -jar /opt/tika/tika-server-standard-3.3.2.jar -c /opt/tika/tika-config.xml

…

So, how can we use this knowledge on the OPA side? In other words, how can we call this Tika API on upload of a document in OPA? Yes, that’s a challenging question…right! Well, trust me…it’s also a challenging answer!

Well, the first part of the answer is easy. When we have a ‘Case’ entity with a ‘Contents’ building block, we can add a ‘Rule’ building block on the relative ‘Contents’ entity which (on the creation event) will start a BPM. Once we’re in the BPM space, we can do several service calls. But what would these be?

First, we need to get our hands on the ‘Content’ instance! This requires a ‘Web Service’ building block on the relative ‘Contents’ entity of the ‘Case’ entity. When exposing the ‘Read’ operation AND creating a supportive service container of type ‘Application Server’ in the ‘System Resource Manager’ artifact, you can do a call like this (passing in the ‘ItemId’ of an uploaded document in the UI!):

1
2
3
4
5
6
7
8
9
10
<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <ReadContents xmlns="http://schemas/opa_tipsgeneric/case.Contents/operations">
      <ns0:Contents-id xmlns:ns0="http://schemas/opa_tipsgeneric/case.Contents">
        <!--<ns0:ItemId>0800270410afa1f1a6eb4bf51e748acb.1</ns0:ItemId>-->
        <ns0:ItemId1>0800270410afa1f1a6eb4e7ff3242a3e.1.2</ns0:ItemId1>
      </ns0:Contents-id>
    </ReadContents>
  </SOAP:Body>
</SOAP:Envelope>

This is the response of that call where the StorageTicket element is the most important:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
<data>
  <wstxns1:ReadContentsResponse xmlns:wstxns1="http://schemas/opa_tipsgeneric/case.Contents/operations" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <wstxns2:Contents xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wstxns1="http://schemas/opa_tipsgeneric/case.Contents/operations" xmlns="http://schemas/opa_tipsgeneric/case.Contents" xmlns:wstxns2="http://schemas/opa_tipsgeneric/case.Contents">
      <Contents-id xmlns="http://schemas/opa_tipsgeneric/case.Contents">
        <Id>1</Id>
        <Id1>2</Id1>
        <ItemId>0800270410afa1f1a6eb4bf51e748acb.1</ItemId>
        <ItemId1>0800270410afa1f1a6eb4e7ff3242a3e.1.2</ItemId1>
      </Contents-id>
      <ParentFolder xmlns="http://schemas/opa_tipsgeneric/case.Contents">
        <Contents-id>
          <Id>1</Id>
          <Id1>1</Id1>
          <ItemId>0800270410afa1f1a6eb4bf51e748acb.1</ItemId>
          <ItemId1>0800270410afa1f1a6eb4e7ff3242a3e.1.1</ItemId1>
        </Contents-id>
      </ParentFolder>
      <wstxns3:ContentTemplate xmlns="http://schemas.opentext.com/bps/entity/buildingblock/contenttemplate" xmlns:wstxns3="http://schemas.opentext.com/bps/entity/buildingblock/contenttemplate">
        <ContentGroup xmlns="http://schemas.opentext.com/bps/entity/buildingblock/contenttemplate">08002704-10af-a1f1-a6eb-505357810b71</ContentGroup>
        <TemplateId xmlns="http://schemas.opentext.com/bps/entity/buildingblock/contenttemplate">08002704-10af-a1f1-a6eb-505357928b71</TemplateId>
        <ContentType xmlns="http://schemas.opentext.com/bps/entity/buildingblock/contenttemplate">FILE</ContentType>
      </wstxns3:ContentTemplate>
      <wstxns4:File xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0" xmlns:wstxns4="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">
        <FileName xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">contract.pdf</FileName>
        <FileSize xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">51744</FileSize>
        <StorageTicket xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">{"documentURL":"/documentstore/opa_tips/opa_tipsgeneric/case/1/Contents/Default/contract.pdf","documentName":"contract.pdf","documentId":"08002704-10af-a1f1-a6eb-6dea72de1e4a","linked":false,"businessWorkspaceId":null,"dctmDocType":null,"versionInfo":null,"signatureStatus":"EMPTY"}</StorageTicket>
        <MimeType xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">application/pdf</MimeType>
        <LockInfo xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0" xsi:nil="true" />
        <LastModified xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/file/1.0">2026-08-19T09:09:43Z</LastModified>
      </wstxns4:File>
      <wstxns5:Title xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/title/1.0" xmlns:wstxns5="http://schemas.opentext.com/entitymodeling/buildingblocks/title/1.0">
        <Value xmlns="http://schemas.opentext.com/entitymodeling/buildingblocks/title/1.0">contract.pdf</Value>
      </wstxns5:Title>
    </wstxns2:Contents>
  </wstxns1:ReadContentsResponse>
</data>

So, we’re focusing on this JSON value from the StorageTicket element above AND specifically the documentURL:

1
2
3
4
5
6
7
8
9
10
{
    "businessWorkspaceId": null,
    "dctmDocType": null,
    "documentId": "08002704-10af-a1f1-a6eb-6dea72de1e4a",
    "documentName": "contract.pdf",
    "documentURL": "/documentstore/opa_tips/opa_tipsgeneric/case/1/Contents/Default/contract.pdf",
    "linked": false,
    "signatureStatus": "EMPTY",
    "versionInfo": null
}

Why the URL? Well, that’s where the document is located in the storage! In our case the OPA storage which you can view with the CMC tool (see also an example here at the end!)

Let’s continue as I don’t want to move too far from the topic this time…

We can reuse that URL in the GetDocument webservice. It’s part of the documentstore method set of the platform, so this requires us to add a runtime reference in our project to make it usable in our BPM, which you need to create. In the BPM we can extract that value from the JSON like this:

1
2
3
4
5
substring-before(
    substring-after(
        ns2:ReadContentsOutput/ns2:ReadContentsResponse/ns3:Contents/ns5:File/ns5:StorageTicket/text(), 'documentURL":"'
    ), '","documentName"'
)

Eventually, you can make a call like this:

1
2
3
4
5
6
7
8
<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <GetDocument xmlns="http://schemas.cordys.com/documentstore/default/1.0">
      <DocumentURL fetchContent="true">/documentstore/opa_tips/opa_tipsgeneric/case/1/Contents/Default/contract.pdf</DocumentURL>
      <FetchSetting>inline</FetchSetting>
    </GetDocument>
  </SOAP:Body>
</SOAP:Envelope>

Notes on the fetch setting:

  • inline returns Base64 content
  • file saves it on the server at /opt/opentext/ProcessAutomation/defaultInst/content/downloadcontent/
  • default is the same as file

This is the response call (with the DocumentContent element containing {Base64} content!):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<data>
  <GetDocumentResponse 
          xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/" 
          xmlns="http://schemas.cordys.com/documentstore/default/1.0">
    <DocumentName>contract.pdf</DocumentName>
    <Description>contract.pdf</Description>
    <FileSize>51744</FileSize>
    <Id>08002704-10af-a1f1-a6eb-6dea72de1e4a</Id>
    <DocumentContent isLocation="false">{Base64}</DocumentContent>
    <Folder>documentstore/opa_tips/opa_tipsgeneric/case/1/Contents/Default</Folder>
    <Properties>
      <MimeType>application/pdf</MimeType>
      <LastModified>2026-08-19T09:09:43</LastModified>
      <Property>
        <Name>ItemId</Name>
        <Value>0800270410afa1f1a6eb4e7ff3242a3e.1.2</Value>
      </Property>
      <Property>
        <Name>RequestUri</Name>
        <Value>http://192.168.56.102:8080/</Value>
      </Property>
      <Property>
        <Name>FileName</Name>
        <Value>contract.pdf</Value>
      </Property>
    </Properties>
  </GetDocumentResponse>
</data>

From a BPM perspective, you can convert Base64 into bytes like this:

1
2
3
4
5
org.apache.commons.codec.binary.Base64.decodeBase64(
    string(
        ns8:GetDocumentResponse/ns8:GetDocumentResponse/ns8:DocumentContent/text()
    )
)

BUT it will show back in a String value [B@21cccbbf, so be aware what you’re doing here before sending it to Tika over the HTTP connector (via a PUT request). It will not work!

So, we need to figure out a PUT HTTP request over the HTTP Connector AND pass a Base64 string as binary bytes!?!? Sure, whatever you want! However, the HTTP connector does not support such a request from what we’ve eXperienced in the past! However, what we can do is create a small extension on the RestRequestHandler in the form of a new class BinaryRestRequestHandler! NICEEEE…What will it look like? Well, as simple as below; sending back a decoded Base64 String.
How to implement this on your own? Read here…

1
2
3
4
5
6
7
public class BinaryRestRequestHandler extends RestRequestHandler {

    @Override
    protected byte[] getPostData(int requestNode, IServerConnection connection) throws HandlerException {
        return Base64.decodeBase64(Node.getData(requestNode));
    }
}

I eventually have a deployed JAR file on the server at /opt/opentext/ProcessAutomation/defaultInst/custom/jad_tika.jar

From an HTTP connector (having that JAR in the JRE config classpath!), we can now implement it all like this; starting with the config.xml for the HTTP service container (saved in the XML store).
Read about it here!

1
2
3
4
5
6
7
<configurations xmlns="http://httpconnector.opentext.com/1.0/configuration">
  <connections>
    <connection id="tika">
      <url>http://localhost:9998</url>
    </connection>
  </connections>
</configurations>

For the HTTP connector, we can create a new custom webservice ws_tika (with interface wsi_tika, namespace http://schemas/opa_tipsgeneric/ws, and operation parse_doc) that has an implementation like this (depending on the HTTP application connector as runtime reference to your project!):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<implementation 
        xmlns:c="http://schemas.cordys.com/cws/1.0" 
        xmlns="http://httpconnector.opentext.com/1.0/implementation" 
        xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/" 
        type="HTTP">
  <connection-id>tika</connection-id>
  <uri>/meta</uri>
  <http-method>PUT</http-method>
  <request-handler class="nl.bos.BinaryRestRequestHandler">
    <req-headers>
      <header name="Content-Type" type="xpath">./mime-type</header>
    </req-headers>
    <root-xpath>./document-content</root-xpath>
  </request-handler>
  <response-handler class="nl.bos.SafeRestResponseHandler" />
  <valid-response-code>200</valid-response-code>
  <namespaces />
</implementation>

We’ll get back on that SafeRestResponseHandler class!

With this as a final SOAP request (where you eventually pass in real values from a BPM perspective!):

1
2
3
4
5
6
7
8
<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <parse_doc xmlns="http://schemas/opa_tipsgeneric/ws">
      <document-content>{GetDocumentResponse.DocumentContent}</document-content>
      <mime-type>{ReadContentsResponse.File.MimeType}</mime-type>
    </parse_doc>
  </SOAP:Body>
</SOAP:Envelope>

Done, with all the thinking/designing…Let’s put it to work! Well, not me…YOU! The BPM process is a simple 3-step process calling ReadContents, GetDocument, and parse_doc with simple source/target mappings!

My eventual response on parse_doc looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
<data>
  <parse_docResponse xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/" xmlns="http://schemas/opa_tipsgeneric/ws">
    <pdf_unmappedUnicodeCharsPerPage>0</pdf_unmappedUnicodeCharsPerPage>
    <pdf_PDFVersion>1.7</pdf_PDFVersion>
    <pdf_docinfo_title>contract.txt - Notepad</pdf_docinfo_title>
    <pdf_hasXFA>false</pdf_hasXFA>
    <access_permission_modify_annotations>true</access_permission_modify_annotations>
    <X-TIKA_Parsed-By-Full-Set>org.apache.tika.parser.DefaultParser</X-TIKA_Parsed-By-Full-Set>
    <X-TIKA_Parsed-By-Full-Set>org.apache.tika.parser.pdf.PDFParser</X-TIKA_Parsed-By-Full-Set>
    <dc_creator>Antal</dc_creator>
    <pdf_num3DAnnotations>0</pdf_num3DAnnotations>
    <dcterms_created>2021-02-01T13:57:05Z</dcterms_created>
    <language>ca</language>
    <dcterms_modified>2021-02-01T13:57:05Z</dcterms_modified>
    <dc_format>application/pdf; version=1.7</dc_format>
    <pdf_overallPercentageUnmappedUnicodeChars>0.0</pdf_overallPercentageUnmappedUnicodeChars>
    <access_permission_fill_in_form>true</access_permission_fill_in_form>
    <pdf_docinfo_modified>2021-02-01T13:57:05Z</pdf_docinfo_modified>
    <pdf_hasCollection>false</pdf_hasCollection>
    <pdf_encrypted>false</pdf_encrypted>
    <dc_title>contract.txt - Notepad</dc_title>
    <pdf_containsNonEmbeddedFont>false</pdf_containsNonEmbeddedFont>
    <Content-Length>51744</Content-Length>
    <pdf_hasMarkedContent>false</pdf_hasMarkedContent>
    <Content-Type-Override>application/pdf</Content-Type-Override>
    <pdf_ocrPageCount>0</pdf_ocrPageCount>
    <Content-Type>application/pdf</Content-Type>
    <access_permission_can_print_faithful>true</access_permission_can_print_faithful>
    <pdf_docinfo_creator>Antal</pdf_docinfo_creator>
    <pdf_producer>Microsoft: Print To PDF</pdf_producer>
    <pdf_totalUnmappedUnicodeChars>0</pdf_totalUnmappedUnicodeChars>
    <access_permission_extract_for_accessibility>true</access_permission_extract_for_accessibility>
    <access_permission_assemble_document>true</access_permission_assemble_document>
    <xmpTPg_NPages>1</xmpTPg_NPages>
    <pdf_hasXMP>false</pdf_hasXMP>
    <pdf_charsPerPage>3631</pdf_charsPerPage>
    <access_permission_extract_content>true</access_permission_extract_content>
    <access_permission_can_print>true</access_permission_can_print>
    <X-TIKA_Parsed-By>org.apache.tika.parser.DefaultParser</X-TIKA_Parsed-By>
    <X-TIKA_Parsed-By>org.apache.tika.parser.pdf.PDFParser</X-TIKA_Parsed-By>
    <access_permission_can_modify>true</access_permission_can_modify>
    <pdf_docinfo_producer>Microsoft: Print To PDF</pdf_docinfo_producer>
    <pdf_docinfo_created>2021-02-01T13:57:05Z</pdf_docinfo_created>
    <pdf_containsDamagedFont>false</pdf_containsDamagedFont>
  </parse_docResponse>
</data>

However, before I got this to work, I needed to fix some Caused by struggles…

First I got this error: Caused by: com.eibus.xml.nom.XMLException: Namespace prefix pdf on unmappedUnicodeCharsPerPage is not defined
pdf:unmappedUnicodeCharsPerPage is a real Tika PDF metadata field. Tika intentionally uses metadata keys such as pdf:PDFVersion, pdf:charsPerPage, xmpTPg:NPages, etc. Our beloved OPA platform apparently sees pdf:unmappedUnicodeCharsPerPage and interprets pdf: as an XML namespace prefix…AHA! But in Tika metadata it is just part of the metadata key…#BUMMER

After a first fix (Adding header Accept: text/plain to the request), I got this error: Caused by: java.io.IOException: Server returned HTTP response code: 406 for URL: http://localhost:9998/meta
Nice header-workaround, but this will only work for a single metadata field; like a call to /meta/Content-Type!

Then (on a first draft version of class SafeRestResponseHandler), I got this error: Caused by: com.eibus.xml.nom.XMLException: Failed to parse QName 'pdf:docinfo:title'
Apparently two colons are possible!? #BUMMER

Finally, I had a free evening to produce (with a little AI-help) this piece of code which works splendidly (even respecting the JSON array values like X-TIKA_Parsed-By!):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
package nl.bos;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Iterator;
import java.util.Map;

import com.eibus.util.logger.CordysLogger;
import com.eibus.xml.nom.Document;
import com.eibus.xml.nom.Node;
import com.eibus.xml.nom.XMLException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.opentext.applicationconnector.httpconnector.IHTTPObject;
import com.opentext.applicationconnector.httpconnector.impl.ContentTypeUtil;
import com.opentext.applicationconnector.httpconnector.impl.RestResponseHandler;
import com.opentext.applicationconnector.httpconnector.management.PerformanceCounters;
import com.opentext.applicationconnector.httpconnector.utils.json.Transformer;

public class SafeRestResponseHandler extends RestResponseHandler {

    private static final CordysLogger LOG = CordysLogger.getCordysLogger(SafeRestResponseHandler.class);

    private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();

    @Override
    protected int convertHttpResponseToXml(IHTTPObject httpObject, Document doc) throws IOException, XMLException {
        //Keep standard OpenText behaviour for non-JSON responses.
        if (!ContentTypeUtil.isContentTypeJson(httpObject.getContentType())) {
            return super.convertHttpResponseToXml(httpObject, doc);
        }

        byte[] responseData = httpObject.getResponseData();
        if (responseData == null || responseData.length == 0) {
            return 0;
        }

        if (LOG.isDebugEnabled()) {
            LOG.debug("Original JSON response:\n" + new String(responseData, StandardCharsets.UTF_8));
        }

        //Sanitize JSON field names BEFORE OpenText converts the JSON into XML.
        byte[] sanitizedJson = sanitizeJson(responseData);
        if (LOG.isDebugEnabled()) {
            LOG.debug("Sanitized JSON response:\n" + new String(sanitizedJson, StandardCharsets.UTF_8));
        }

        //Now use the normal OpenText JSON -> XML transformer.
        byte[] xml = transformJsonToXml(sanitizedJson);
        if (LOG.isDebugEnabled()) {
            LOG.debug("Generated XML response:\n" + new String(xml, StandardCharsets.UTF_8));
        }

        int xmlNode = doc.load(xml);
        
        //Preserve StandardResponseHandler parent behaviour.
        int parentNode = defineParentNode(doc);

        if (parentNode == 0) {
            parentNode = xmlNode;
        } else {
            Node.appendToChildren(xmlNode, parentNode);
        }

        return parentNode;
    }

    private byte[] sanitizeJson(byte[] json) throws IOException {
        JsonNode root = OBJECT_MAPPER.readTree(json);
        JsonNode sanitizedRoot = sanitizeNode(root);
        return OBJECT_MAPPER.writeValueAsBytes(sanitizedRoot);
    }

    private JsonNode sanitizeNode(JsonNode node) {
        if (node.isObject()) {
            ObjectNode sanitizedObject = OBJECT_MAPPER.createObjectNode();
            Iterator<Map.Entry<String, JsonNode>> fields = node.fields();

            while (fields.hasNext()) {
                Map.Entry<String, JsonNode> field = fields.next();

                String originalName = field.getKey();
                String sanitizedName = sanitizeFieldName(originalName);

                if (LOG.isDebugEnabled() && !originalName.equals(sanitizedName)) {
                    LOG.debug("Sanitized JSON field name: " + originalName + " -> " + sanitizedName);
                }
                sanitizedObject.set(sanitizedName, sanitizeNode(field.getValue()));
            }

            return sanitizedObject;
        }

        if (node.isArray()) {
            ArrayNode sanitizedArray = OBJECT_MAPPER.createArrayNode();
            for (JsonNode child : node) {
                sanitizedArray.add(sanitizeNode(child));
            }
            return sanitizedArray;
        }

        //Strings, booleans, numbers and null: leave completely untouched.
        return node;
    }

    private String sanitizeFieldName(String name) {
        //Keep XML-friendly characters. ':' and other invalid/special characters become '_'.
        String sanitized = name.replaceAll("[^A-Za-z0-9_.-]", "_");

        //XML element names cannot start with a number, dot or hyphen.
        if (sanitized.isEmpty() || !Character.isLetter(sanitized.charAt(0)) && sanitized.charAt(0) != '_') {
            sanitized = "_" + sanitized;
        }

        return sanitized;
    }

    private byte[] transformJsonToXml(byte[] json) {
        try (PerformanceCounters.CounterWrapper wrapper = getPerformanceCounters().updateJsonToXmlTransformationCounter()) {
            return Transformer.getInstance().jsonToXml(new ByteArrayInputStream(json), "result");
        }
    }
}

I know…it’s not my first step to customize/extend code AND it was also not expected to be needed. However, when things don’t work as expected, you need to make the best out of it. Well, we learned at least a lot on this implementation which we can reuse on other projects again.

That’s a valuable “DONE” where we see a totally unexpected (and underrated) tool doing its magic on our valuable content in OPA. I don’t know any more how I got to this tool, but I just bumped into it during some browsing. Now you see the importance of doing R&D! It brings new tools, new perspectives, and new ways of thinking. Grab yourself a weekend to do the same as it will enhance your knowledge greatly. See you, next week, in another topic about “OpenText Process Automation Tips”.

Don’t forget to subscribe to get updates on the activities happening on this site. Have you noticed the quiz where you can find out if you are also “The Process Automation guy”?