/ Management  

Master the game; Using Gmail as the ultimate email service container for our beloved platform

Hi there “Process Automation” fans,

Welcome to a new installment of “Process Automation” tips.

What is it with mail these days? We always complain about getting too much mail, but the first thing we always directly need to implement is mail notifications!? Is it the world up-side-down? People not trusting the system? Just a bad behavior of humanity? A managers’ choice? You tell me in the comments…

For your perspective: Mail is unstructured data, and you need to restructure it with proper metadata for easy searching and finally better AI output…If you’re already in the future!

In projects, I always discourage to use a mail service container for sending out all kinds of mail messages from entity Rules or BPM activities. I do agree on sending out mail from an administration perspective when ^%$@%$!$ hits the wall, but for regular end-users there is no need. It’s far better to have them logged in into a portal where all messages are structured and where communication funnels to one central location. Large companies use exactly this mindset to inform their customer base and we as developers should advise the same from our beloved “Process Automation” platform.

The platform has sufficient connectors to send out events over HTTP (with SOAP/JSON), JMS (publish/subscribe), or FTP (although old).

But because I always see OpenText demos shining with Gmail as mail provider, it’s about time we’ll just make sure we have a post available that makes it work. It’s not a hard thing…if you know how to move from A to Z!

For your reference:

  • I used smtp4dev in this post before! Nice tool for simply sending out an e-mail on a local running SMTP instance.
  • Set up your own mail server with these posts over ‘postfix’ and ‘dovecot’ (both Linux!)
  • In the past I used Mailtrap as mail server for development purposes (not tried with “Process Automation”!)
  • I also see Mailslurper passing by as alternative (also not tried with “Process Automation”!)

Let’s continue the post on Gmail to send out mail, but also read mail!

Let get right into it…

Let’s first set up a Gmail account…Well, I’m not going to explain this, that’s a task on your own! 🥳 Make sure you can log in with the credentials, send a mail out, and receive a mail back. Once ready…continue your read.

Time to create ourselves a brand-new service container of type ‘E-mail’ from the ‘System Resource Manager’ artifact in your organization; Use this as input:

  • Connector type: E-mail
  • Group name: sg_email
    • Select web service interface: Method Set Email
  • Container name: sc_email
    • Assign it to the OS (recommended by OpenText for all service containers)
    • Let it start automatically
  • See the images/XML parts below for the last configuration steps!

Mail Server (you can leave the ‘MIME Options’ as is…They’re not in the screenshot!)

gmail_001

If you want to have it secure over SSL/TLS do it like this:
gmail_002

Mailboxes (which we’ll update later!)

1
<emailboxes xmlns="http://schemas.cordys.com/1.0/email/configuration" ></emailboxes>

Database Configuration (just reusing the existing DB)

gmail_003

Key managers (which you can leave like this and which I also leave out of scope for this post):

1
<keymanagers xmlns="http://schemas.cordys.com/1.0/email/configuration" ></keymanagers>

Other resources:

Send out an E-mail

We can do it the hard way over entity rule craftsmanship, but let’s do it quick and dirty with a service call via the ‘Web Service Interface Explorer’ artifact!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <SendMail xmlns="http://schemas.cordys.com/1.0/email">
      <to>
        <address>
          <emailAddress>antal@bos-ictservices.nl</emailAddress>
          <displayName>Antal</displayName>
        </address>
      </to>
      <subject>Hello world</subject>
      <body type="normal">Hello world...</body>
      <from>
        <displayName>OPA</displayName>
        <emailAddress>OPA@gmail.com</emailAddress>
      </from>
    </SendMail>
  </SOAP:Body>
</SOAP:Envelope>

Watch out for the correct service call as we also have SendMailMIME, sendEmail and sendEmailFromTemplate!

After invoking the service call, it all looks fine, but no mail arrives in my mailbox!? Well, this should be an authentication thing as I never provided any Gmail-credentials (yet!) as the checkmark tells us in the service container (see screenshot above on the Authentication required setting for outgoing mail). Looking at the log of the new mail service container it tells me about this detail on that first message call:

1
2
3
An error occurred while processing the SOAP request: 
Cannot invoke "String.isEmpty()" 
because "com.eibus.applicationconnector.email.MailMethod.sUserName" is null

gmail_004

Ohw…This logging shows old stuff first!?…Don’t shoot the messenger! 🙃

So far all as expected…let’s continue and set credentials for our outgoing mail via a SetProfile call:

1
2
3
4
5
6
7
8
9
10
11
12
13
<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP:Body>
    <SetProfile xmlns="http://schemas.cordys.com/1.0/email">
      <displayName>OPA</displayName>
      <mailId>OPA</mailId>
      <!--Dont't use the real password, but use an 
      "apppassword" https://myaccount.google.com/apppasswords-->
      <password>xxxx xxxx xxxx xxxx</password>
      <!--Use this as "username" to authenticate against Google itself!-->
      <userId>OPA@gmail.com</userId>
    </SetProfile>
  </SOAP:Body>
</SOAP:Envelope>

Have also a read in the administration manual of the platform within section ‘Configuring the E-mail connector’.
…
The “apppassword” is a Google security thing (only for 2FA enabled accounts…the default today!) and is the only way (after my too long R&D struggle session) to make it work! Googles’ “Less secure apps” setting is a deprecated feature, and “Passkeys and security keys” is (I think) the new way, but not for now!
…
Read also this interesting thread about the topic!

Right, so we can now send a mail (I tried again, and it works) via the service…what’s next?

Receive an E-mail

How about receiving a mail!? Well, receiving is a simple polling mechanism based on a mail configuration; We did it before, but let’s start small with a basic emailboxes configuration in the mail service container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<emailboxes xmlns="http://schemas.cordys.com/1.0/email/configuration">
  <emailbox>
    <name>Gmail</name>
    <!--Use this as "username" to authenticate against Google itself!-->
    <username>OPA@gmail.com</username>
    <!--Dont't use the real password, but use an 
    "apppassword" https://myaccount.google.com/apppasswords-->
    <password>xxxx xxxx xxxx xxxx</password>
    <folders>
      <!--Make sure you have this as new label available within 
      your Gmail mailbox and apply a test mail to it-->
      <folder>OPA_TEST</folder>
    </folders>
  </emailbox>
</emailboxes>

Don’t worry about the plain password here; It magically removes itself during restart of the service container and saved (encrypted) in the configuration of CARS/OpenLDAP:
gmail_005
…
Read about this JXplorer tool here
…
If Your account is not enabled for IMAP use change it in your Gmail settings:
gmail_006
If you really can’t use IMAP; Try it over POP3 with corresponding host/port: pop.gmail.com:995
…
Be aware that POP3 will remove the mail from the server where IMAP leaves it (which is also my preferred way of doing things, but it all depends!). Also be aware that POP3 will always (be default) check the INBOX-folder whatever else you use in the emailboxes configuration of the mail container…that was at least my experience trying it out.

When all is fine the service container will start with a green icon again and will start polling the OPA_TEST folder/label of your Gmail account. You can have a view enabling the log level for the email service container like this:

gmail_007

These messages will pass by in sudo tail -999f /opt/opentext/ProcessAutomationCE/defaultInst/Logs/opa_tips#sg_email#sc_email.xml

1
2
3
4
[Gmail] Polling starts now, monitoring 1 folders at Mon Jan 06 12:10:32 CET 2025
[Gmail] Folder OPA_TEST contains 1 messages
[Gmail] Found 0 triggers for folder OPA_TEST
[Gmail] Closing email connection

So, it does “see” my inbox message already, but it doesn’t do anything with it! Indeed correct, but that’s a trigger we can create in the emailboxes configuration with a minimal implementation like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<emailboxes xmlns="http://schemas.cordys.com/1.0/email/configuration">
  <emailbox>
    <name>Gmail</name>
    <username>OPA@gmail.com</username>
    <!--You can leave it empty once it set the first time!-->
    <password/>
    <folders>
      <folder>OPA_TEST</folder>
    </folders>
    <triggers>
      <trigger appliesTo="OPA_TEST">
        <name>AllFolders</name>
        <rules>
          <rule section="SUBJECT">
            <pattern type="REGEX">
              <value>Hello world</value>
            </pattern>
          </rule>
        </rules>
        <message>
          <method>ExecuteProcess</method>
          <timeout>60000</timeout>
          <namespace>http://schemas.cordys.com/bpm/execution/1.0</namespace>
          <user>cn=opadev@opa,cn=organizational users,o=opa_tips,cn=cordys,cn=defaultInst,o=mydomain.com</user>
          <sync>true</sync>
          <input>
            <type>definition</type>
            <receiver>nl-bos-general/bpms/bpm_dummy</receiver>
            <source>Run from Mail trigger</source>
          </input>
        </message>
      </trigger>
    </triggers>
  </emailbox>
</emailboxes>

That bpm_dummy BPM looks like this in my project:
gmail_008

What does this do when we restart the service container? Well, as long as the subject of my mail within folder ‘OPA_TEST’ does not contain ‘Hello world’ it will not do anything! Well, you’re partly correct as what I see now is that every message found in the folder gets a ‘Read’ flag on the server! I partly agree as our “poller” reads the message eventually, but it’s an interesting finding! This also means we can probably check the ‘Read’ state of the message and act on it (that’s for another time)

What does it do when it finds a message containing ‘Hello world’? Well, the trigger goes off (as expected!) AND the message is removed/archived (or better “expunged”) from the inbox! That’s also interesting as IMAP is designed to leave the message on the server until the client tells otherwise (if I understand and remember correctly). Sound to me like our mail container “poller” sends this information directly back to get rid of the message server side…Well, it is what it is.

This is what I see passing by in the logging:

1
2
3
4
5
6
7
8
[Gmail] Polling starts now, monitoring 1 folders at Mon Jan 06 12:10:32 CET 2025
[Gmail] Folder OPA_TEST contains 1 messages
[Gmail] Email Gmail_<887260548.5.1736159482162@opa.mydomain.com> matches trigger AllFolders.
Time taken for reading the email content from mail server is 117 ms, for messageId - <887260548.5.1736159482162@opa.mydomain.com>, subject - Hello world, size - 933 bytes
Going to add the context to the database with id 08002760-7520-a1ef-b3ac-36ebbf4d02cb and logical ID Gmail_<887260548.5.1736159482162@opa.mydomain.com>.
Removing email message with message-id: <887260548.5.1736159482162@opa.mydomain.com>
Time taken for processing email from mailbox is 1,088 ms, for messageId - <887260548.5.1736159482162@opa.mydomain.com>
[Gmail] Closing email connection

Be aware that the trigger will only go off once! When the message saves in the database (as the logs tells us), you can do whatever you want with the message in Gmail (like reattaching it to the folder ‘OPA_TEST’) but it will not fire the trigger again! A new mail matching the rule will fire a new trigger!

Don’t forget to also watch the PIM for instances of this BPM!

Final Q/A after all the R&D

❓ Where does ‘SetProfile’ stores its password and how?

After some remote debugging we saw this passing by: UserProfile.encryptAndEncode(this.sUserId, sPassword);. Remember?

Eventually we see a save into CARS/OpenLDAP at this location (so, that’s per user account!)

gmail_009

❓ Where is the mail message saved in the database?

Well, how about the ‘email’ table: select * from email. With a result (export as JSON for your readability) like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
 "table": "email",
 "rows":
 [
  {
   "email_id": "08002760-7520-a1ef-b3d8-ec3cc3a9a543",
   "emailbox": "08002760-7520-a1ef-b306-cf7d30aa8aea",
   "logical_id": "Gmail_<1274887064.3.1736159102776@opa.mydomain.com>",
   "mail_size": 933,
   "processing_container": "cn=sc_email,cn=sg_email,cn=soap nodes,o=opa_tips,cn=cordys,cn=defaultInst,o=mydomain.com",
   "processing_status": "COMPLETED",
   "create_date": "2025-01-10 15:00:54+01",
   "status_change_date": "2025-01-10 15:00:57+01",
   "complete_date": "2025-01-10 15:00:57+01",
   "message_id": "<1274887064.3.1736159102776@opa.mydomain.com>",
   "send_date": "2025-01-06 10:25:02+01",
   "receive_date": "2025-01-06 10:25:04+01",
   "from_address": "OPA <OPA@gmail.com>;",
   "to_address": "Antal <antal@bos-ictservices.nl>;",
   "subject": "Hello world",
   "status_information": "Request took 1747ms",
   "trigger_definition": "<trigger>...</trigger>",
   "content": "..."
  }
 ]
}

And guess what…When you remove the entry from the table and set the mail back to its folder the trigger does go off again!! 😎

Nicely “DONE”, we went beyond our scope but with again some valuable lessons and examples of managing mail from a Gmail provider. I almost gave up but found the light again with the Gmail “apppassword” settings. We can now send mail via Gmail, and we can even fire a BPM instance once a new mail fills the configured trigger. Have a recap on this mailing journey…For me, it’s time for weekend; Cheers! 🍺

Don’t forget to subscribe to get updates on the activities happening on this site. Have you noticed the quiz where you find out if you are also “The Process Automation guy”?