Hi there AppWorks fans,

Welcome to a new installment of AppWorks tips.

Three weeks ago (before “OT World Las Vegas”) we saw an easy clean-up with an un-deploy and re-deploy of the solution. From what we experienced (in my current project) this was fine for TST and ACP environments (which was our target). The only thing this procedure is not cleaning up are the BPM instances, lifecycle (or case) instances, task instances in the inbox, and notification instances. So, in our organization we could still open the inbox of a user account and see BPM tasks passing by. Yes, I know we have a post available on cleaning BPM instances with the ‘Process Archival Manager’ artifact. This already cleans the process instances, but I didn’t check if it also cleans the inbox items for a user. During the whole procedure, we saw on interesting chapter passing by in the administration manual with the name “Deleting the runtime content”. This chapter describes exactly the missing pieces on our cleaning activities. Time to have a look at the ‘archiveframework’ of the platform which cleans the rest of our mess…

Let get right into it…

Great, let’s first start with a nice solution with these items in place:

• Entity (with just a ‘Name’ property and default generated BBs)
• Entity lifecycle (so generate case instances in the CIM)
  • The state should start an activity with an attached functional role
  • Apply this role also to your current user!
• Entity action rule (starting a BPM for instances in the PIM)
  • The BPM gets a manual task (for retrieving inbox items; via a related lifecycle task entity layout)

Things like roles (pushed to OTDS and consolidated back to AWP), entity CRUD services and related service containers are untouched in my development-situation; I leave them out of scope for this post.

…The build on such a solution I leave with you…

Make sure you have instances for all entry points available, so you have solid input for the next section where we dive into the ‘archiveframework’. So, what instances do I currently have in my (from scratch) solution?

• 5 ‘Project’ entity instances

• 5 related lifecycle instance in ‘Case Instance Manager’

• 5 initialized ‘long-lived’ BPMs in the ‘Process Instance Manager’

• 10 tasks in my inbox (5 from the lifecycle, 5 from the BPM activity)

• Two roles (1 ‘functional’, 1 ‘application’) in the identity package (pushed from AWP to OTDS; consolidated back from OTDS to AWP). I also created one ‘internal’ role, but this one is only platform related and is not getting a sync by default! (the roles are out of scope for this post…Have a look at this post for more information on pushing/consolidating roles)

• One appserver_service_container service container with the CRUD service interfaces of my entity (creating the service container is out of scope for this post…This side has plenty of sample available; have a search).

FYI: The documentation is also mentioning cleaning ‘Notifications’; I leave these out of scope as they are part of the ‘older’ part of the platform. We can create notifications, but we build it with entity modelling these days.

When everything is in place, it’s time for the next step…

Init the “Archive framework”

First a re-thought…I was creating all the above scenarios with the intention to package it and to install it all to a TST server like the previous post. Only, why not cleaning it from a developer perspective this time? We as developers also like to have a clean sandbox to play in.

…WAIT…First scroll to the bottom and read the lessons learned first! In a review of this post I first aborted the PIM/CIM instances and then removed the solution from app/admin.

…

So, the first thing to do is removing our solution from the /app/admin layer of the platform (which was not a smart move to do after my review!). This will remove the entity instances, but will it remove all the rest?

After this ‘Delete’ action, I clean my build output from a workspace perspective, and I do a re-publish of the project. At this moment I have a clean environment available!? NOPE! Only the entity instances are gone, but the lifecycle instances, bpm instances and tasks are still available in the CIM, PIM, and runtime inbox for the user.

What about the roles and the service container? Well, those are still untouched; with also the identity package roles in place from a runtime perspective. Great, time for cleaning the left-overs…

Let’s get ourselves a grip on the ‘archiveframework’ CLI tool…I just go in to it with a straight leg by opening a MobaXTerm session to the VM and give these commands. From another CLI tool we learned it’s best to use the same account as the installation account of AWP.

1
2
3
4
5

whoami #should return the same account from an AWP installation perspective (in my case `sysadmin`); You also create the first organization with it!
export CLASSPATH=$CLASSPATH:/opt/opentext/AppWorksPlatform/defaultInst/components/archiveframework/archivingengine.jar
cd /opt/opentext/AppWorksPlatform/defaultInst/components/archiveframework/bin
sudo chmod +x *.sh
./DataDeletionTool.sh

The classpath export is required as the tool depends on that JAR file.

After execution, I get the message: Cordys.ContentArchivingFramework.Messages.invalidArguments. A valid conclusion as I did not #RTFM…After this manual scanning session, I update the call with some “valid” arguments!?

./DataDeletionTool.sh arg1 arg2

It triggers, but still with missing dependencies…A FEW MOMENTS LATER!

1
2
3

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.el8_6.x86_64/lib/server:/opt/opentext/AppWorksPlatform/defaultInst/lib
export CORDYS_HOME=/opt/opentext/AppWorksPlatform/defaultInst
export PATH=$PATH:$CORDYS_HOME/bin

After all the exporting tricks and running the same command again another valid conclusion is made by the tool: Caused by: java.lang.IllegalArgumentException: cn=sysadmin,cn=organizational users,arg2 does not end with cn=cordys,cn=defaultinst,o=22.3.com

arg2 is indeed not valid…The documentation is describing a valid “Organization DN”!

Let’s try this one: ./DataDeletionTool.sh arg1 o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com

Aha…One step closer to our target with a message: There are no ArchivableDataType documents deployed.…Back to our admin manual, as that’s the part I skipped…😅

Deploy/register an ‘ArchivableDataType’ document

There is some familiar scripting involved in this part of the post…Remember this bash-content-part?

1
2
3
4
5

sudo -u postgres psql
\c appworks_db
\i /opt/opentext/AppWorksPlatform/defaultInst/components/archiveframework/database/createscripts/BPM_CASE_ARCHIVAL_POSTGRES.sql
\i /opt/opentext/AppWorksPlatform/defaultInst/components/archiveframework/database/createscripts/NOTIFICATION_ARCHIVAL_POSTGRES.sql
\q

Yes, my friends…directly copied from this post! So, it’s connected! The next step is creating our first “archive definition” XML file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<ArchivingDefinition xmlns="com.cordys.archiving.framework">
    <FQN>nl.bos.archive.HelloWorldArchivingDefinition</FQN>
    <Description locale="en-US">This ArchivingDefinition is used to delete (skip archival) all finished BPM instances incl. associations</Description>
    <TypeReference>
        <ArchivableDataTypeFQN>com.cordys.archiving.bpm</ArchivableDataTypeFQN>
        <Criteria>
            <BatchSize>10</BatchSize>
            <ArchiveData>false</ArchiveData>
            <Filter>
                <FilterCriteria>
                    <TimeInterval>
                        <Age>0</Age>
                    </TimeInterval>
                    <ArchiveAssociatedInstances>true</ArchiveAssociatedInstances>
                </FilterCriteria>
            </Filter>
        </Criteria>
    </TypeReference>
</ArchivingDefinition>

Don’t forget to add the ‘FilterCriteria’ element…it’s required! During the execution of the ‘ArchivingDefinitionManagerTool’ the XML is validated to an XSD (an XML definition describing how the XML should look like).

I save this file as HelloWorldArchivingDefinition.xml directly in the ‘bin’ directory of the ‘DataDeletionTool’. After this I run the command: ./ArchivingDefinitionManagerTool.sh HelloWorldArchivingDefinition.xml o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com create

Right…An error…again! This time about an ‘access denied’ on a service call. More specific an ‘access denied’ on an XML Store location /Cordys/archiving-framework/ArchivableDataTypes…Let’s fix this via the ‘XMLStore Explorer’ artifact:

Running the command again gives me a final relieve ‘The archiving definition has been created successfully.’ with a change in the XML store:

Leave this data as is…It’s not meant to be touched from this perspective…You can remove it again with the delete flag in the command (replacing the create flag; and don’t forget to provide the FQN instead of the XML definition!)

How about the archiving tables we’ve created? Well, we don’t use them because of the <ArchiveData>false</ArchiveData> element of the archive definition XML file! I leave the flag to you for trying out…I’m on DEV and don’t give a “F$%^” about archiving. I’m glad it’s gone already! 😎

So, preparation is done…back to where we left…

Second try “Archive framework”

Hit the gas with ./DataDeletionTool.sh arg1 o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com…”Access denied” on XML store location /Cordys/archiving-framework/RepositoryTypes! Well, you know the drill now!

Hit the gas again…Error…The ArchivingDefinition with FQN 'arg1' is not found! Off-course, I was wondering when it came with this final “valid” argument!

Switching gears with: ./DataDeletionTool.sh nl.bos.archive.HelloWorldArchivingDefinition o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com

OHWWWW MAMA!! 😍

1
2
3

The archival tool with archiving definition 'nl.bos.archive.HelloWorldArchivingDefinition' has been initiated.
The archival tool executed successfully.
Total Time taken for executing the tool: 3763 ms

Any logging? Yes my fiend…

1
2
3
4
5
6
7
8

# When things go wrong:
tail -999f /opt/opentext/AppWorksPlatform/defaultInst/Logs/ArchivingDefinitionManagerTool.xml
tail -999f /opt/opentext/AppWorksPlatform/defaultInst/Logs/DataDeletionTool.xml

# When you nailed it:
ls -ltr /opt/opentext/AppWorksPlatform/defaultInst/Logs/
cat /opt/opentext/AppWorksPlatform/defaultInst/Logs/ArchivingLog-nl.bos.archive.HelloWorldArchivingDefinition-1660135360144.txt
cat /opt/opentext/AppWorksPlatform/defaultInst/Logs/ArchivingSummary-nl.bos.archive.HelloWorldArchivingDefinition-1660135360144.txt

Looking all nice and steady at my side…Except for one thing! My BPMs are in “Waiting” state, but this is NOT a valid state to clean up based on my XML definition. My entity instances are already removed and the BPMs can get an abortion? from the PIM perspective.

To be clear…I executed this ‘Terminate’ action before removing the solution from /app/admin!

Running the ‘DataDeletionTool’ will eventually pick up my TERMINATED, COMPLETE, and ABORTED BPMs as the archive definition XML is telling us! I’m not sure about the ‘aborted’ state as I don’t have any aborted instances…Make your own conclusion on your own data set.

This is an advanced XML sample for reference:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<ArchivingDefinition xmlns="com.cordys.archiving.framework">
    <FQN>nl.bos.archive.BPMArchivingDefinition</FQN>
    <Description locale="en-US">This AD is used to delete it all specific BPM instances by state</Description>
    <TypeReference>
        <ArchivableDataTypeFQN>com.cordys.archiving.bpm</ArchivableDataTypeFQN>
        <Criteria>
            <BatchSize>10</BatchSize>
            <ArchiveData>false</ArchiveData>
            <Filter>
                <FilterCriteria>
                    <ProcessModels>
                        <Value>nl-bos-gen/bpms/bpm_test</Value>
                    </ProcessModels>
                    <InstanceStatus>
                        <Value>COMPLETE</Value>
                        <Value>TERMINATED</Value>
                        <Value>ABORTED</Value>
                    </InstanceStatus>
                    <TimeInterval>
                        <Age>0</Age>
                    </TimeInterval>
                    <InstantiationUser>
                        <Value>cn=awdev@awp,cn=organizational users,o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com</Value>
                    </InstantiationUser>
                    <ArchiveAssociatedInstances>false</ArchiveAssociatedInstances>
                </FilterCriteria>
            </Filter>
        </Criteria>
    </TypeReference>
</ArchivingDefinition>

More to clean up

Well, my BPMs are gone from the PIM, but what about the lifecycle instances and task instances? An interesting conclusion is that my related BPM tasks in the inbox are also removed already! So, that’s a second job done! 😄 We’re left with the lifecycle instances and corresponding lifecycle task instances.

How about this XML definition:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<ArchivingDefinition xmlns="com.cordys.archiving.framework">
    <FQN>nl.bos.archive.CaseArchivingDefinition</FQN>
    <Description locale="en-US">This AD is used to delete all finished Case instances incl. associations</Description>
    <TypeReference>
        <ArchivableDataTypeFQN>com.cordys.archiving.casemanagement</ArchivableDataTypeFQN>
        <Criteria>
            <BatchSize>10</BatchSize>
            <ArchiveData>false</ArchiveData>
            <Filter>
                <FilterCriteria>
                    <TimeInterval>
                        <Age>0</Age>
                    </TimeInterval>
                    <ArchiveAssociatedInstances>true</ArchiveAssociatedInstances>
                </FilterCriteria>
            </Filter>
        </Criteria>
    </TypeReference>
</ArchivingDefinition>

Installing and running it:

1
2
3
4
5
6
7
8

vi CaseArchivingDefinition.xml
./ArchivingDefinitionManagerTool.sh CaseArchivingDefinition.xml o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com create
./DataDeletionTool.sh nl.bos.archive.CaseArchivingDefinition o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com
ls -ltr /opt/opentext/AppWorksPlatform/defaultInst/Logs/
cat /opt/opentext/AppWorksPlatform/defaultInst/Logs/ArchivingSummary-nl.bos.archive.CaseArchivingDefinition-1660219175960.txt

# Cleaning up (if needed)
./ArchivingDefinitionManagerTool.sh nl.bos.archive.CaseArchivingDefinition o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com delete

Looking all nice and again great at my side…Except for just that one thing! My Case instances are in “NEW” state, but this is NOT a valid state to clean up based on my XML definition. My entity instances are already removed and the Case instances can get an abortion? from the CIM perspective.

Same clarification here…I executed this ‘Terminate’ action before removing the solution from /app/admin!

Running the ‘DataDeletionTool’ will eventually pick up my TERMINATED, COMPLETED, and ABORTED Case instances. Again…not sure about the ‘aborted’ state! I leave it up to you.

This is an advanced XML sample for reference:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

<ArchivingDefinition xmlns="com.cordys.archiving.framework">
    <FQN>nl.bos.archive.CaseArchivingDefinition</FQN>
    <Description locale="en-US">This AD is used to delete it all completed Case instances</Description>
    <TypeReference>
        <ArchivableDataTypeFQN>com.cordys.archiving.casemanagement</ArchivableDataTypeFQN>
        <Criteria>
            <BatchSize>10</BatchSize>
            <ArchiveData>false</ArchiveData>
            <Filter>
                <FilterCriteria>
                    <CaseModel>
                        <Value>nl-bos-gen/entities/project#ef#/bb/Lifecycle/Lifecycle/project</Value>
                    </CaseModel>
                    <InstanceStatus>
                        <Value>COMPLETED</Value>
                        <Value>TERMINATED</Value>
                    </InstanceStatus>
                    <TimeInterval>
                        <Age>0</Age>
                    </TimeInterval>
                    <InstantiationUser>
                        <Value>cn=awdev@awp,cn=organizational users,o=appworks_tips,cn=cordys,cn=defaultInst,o=22.3.com</Value>
                    </InstantiationUser>
                    <ArchiveAssociatedInstances>false</ArchiveAssociatedInstances>
                </FilterCriteria>
            </Filter>
        </Criteria>
    </TypeReference>
</ArchivingDefinition>

The CaseModel element gets special attention! I wasn’t able to figure out the FQN is for my project lifecycle BB. I tried nl-bos-gen/entities/project and also nl-bos-gen/entities/project#ef#/bb/Lifecycle, but after a debug session in the Chrome developer tools I found my purpose:

How about the lifecycle tasks in the inbox? Well, they are magically removed already! That’s it…My environment is clean now, but with some lessons learned…

Lessons learned

• First terminate/abort the BPMs and Case instances before cleaning the entity instances! Why? well I see in the logging missing IDs passing by during my “abort” actions! So, it searches for references.
• BPM has a state COMPLETE; Case has a state COMPLETED!? Don’t shoot the messenger!
• Lifecycle tasks and BPM tasks are removed seamlessly with the described actions.
• I was unable to figure out the case-model FQN for my entity, but found it eventually via a debug session.
• I found a bug…See below!

…

I also found a BUG (I guess; as it runs smoothly after my own fix) in the scripts of the platform during Case instance cleanup. The delete-tool failed over and over again because of a missing function in my Postgres database with the name delete_inactive_case_model_data. A reverse engineering session with decompiling Java code (‘archivingengine.jar’ en ‘bpmengine.jar’) brings me to this call (with a ‘ternary’ operated if-else-statement!):

1
2
3
4
5

CallableStatement = 
    "Oracle".equals(this.dBVendor) ? 
        this.dbConnection.prepareCall("{CALL DELETE_CASE_MODEL_DATA (?)}") : 
        this.dbConnection.prepareCall("{CALL DELETE_INACTIVE_CASE_MODEL_DATA (?)}")
    ;

My dBVender is Postgres which calls a missing in action function with name DELETE_INACTIVE_CASE_MODEL_DATA. Why missing? Well, querying my functions in my appworks_db with HeidiSQL…

1
2
3
4
5

SELECT routine_name
FROM   information_schema.routines
WHERE  routine_type = 'FUNCTION'
       AND routine_schema = 'public'
       AND routine_name LIKE 'delete%';

…does not provide me back with a valid function. I only could find this function in 2 specific scripts in this location on the server: /opt/opentext/AppWorksPlatform/defaultInst/components/bpmengine/database/archival/case. It’s only implemented for ‘MYSQL’ and ‘MSSQL’; for my POSTGRES version, I can only find the function delete_case_model_data.

The solution: Updating file vi /opt/opentext/AppWorksPlatform/defaultInst/components/bpmengine/database/archival/case/CASE_ARCHIVAL_DELETE_POSTGRESQL.sql with a content change like this:

1
2

--CREATE OR REPLACE FUNCTION delete_case_model_data(archive_enable boolean)
CREATE OR REPLACE FUNCTION delete_inactive_case_model_data(archive_enable boolean)

After installing it with the already described bash commands, my delete-tool did its job very well…Finally! 😅

We’ve reached a “DONE” again on an interesting post on cleaning runtime content with the highly valuable ‘archiveframework’ CLI tool. From the previous post we learned about cleaning entity instances. This tool is an addition to the cleaning part in runtime when you are left with BPM instances, lifecycle instances, task instances, and notification instances. So, we learned our lessons again; Have a great clean-up during your own weekend, and I see you next week on another AppWorks Tips. See you later - “Ciao”!

Don’t forget to subscribe to get updates on the activities happening on this site. Have you noticed the quiz where you find out if you are also “The AppWorks guy”?