Andy Orrock | Payment Systems

Illustration of the 15-touch customer interaction

I just concluded a really fun, rewarding Elance engagement in which I commissioned a project to illustrate and animate the 15-touch point-of-sale interaction that I described in this piece.  You can see the results here.  We'll put the results up on our home page later this week.

The results are meant to be self-explanatory, so I won't belabor the point here.

If you liked how this turned out, I recommend you check out my service provider, Light Within Productions (a.k.a., Ace Light).  I had a really great experience on this project - my best Elance experience to date.

Transaction counts at 2010 year-end

Now that 2010 has drawn to a close, I was interested in where we sit in terms of 'transactions processed to date' at our largest acquirer-side payment switch implementation.  Here's the tale of the tape:

Rx:                    576,259,057 [Adjudication]
Financial [Non-PCI]:   395,830,634 [Mostly Loyalty]
Financial [PCI]:     1,293,382,153 [Debit, Credit, ECA, SV]
                     -------------
Total:               2,265,471,844

Even though the Financial PCI implementation has the biggest number, it's only a reflection of it being the first of our three implementations there.  The other two systems are growing at far greater rates.

Related Posts:

PCI Split

An update on our Rx adjudication project efforts

Live with ECA

Implementing MethCheck

 

Interesting EOY Reversal Dynamics

At one of our acquirer-side payment switch implementations, the terminal-generated reversal/void count is exhibiting its annual end-of-year spike:

What's going on here?  This phenomenon is the result of people rushing to this nation-wide retail chain looking to fully use their Flexible Spending or Health Savings Accounts ('FSA/HSA') prior to those benefits expiring at end of year.  Our implementation supports partial approvals of these transactions.  What's happening in these situations is:

1. We format and send the transaction to our client's FSA/HSA gateway (here, the excellent FDR North network).  As part of this transaction, we turn on an indicator asking FDR to inform the card issuer that we're capable of supporting partial approvals.
2. We get an approval back along with the original amount we requested and the amount the issuer approved.
3. We inform our client's store system of the approval amount.
4. In cases where the approved amount is a partial, the store system instructs the associate to ask for another form of payment to cover the remaining amount.
5. At this point, most people provide the funds to complete the sale from another source.  But, in many cases, the customer elects to void the sale or reduce the size of their market basket.  [This is understandable behavior: many people aren't aware of the exact balance of their card; and HSA transactions are complicated by the item qualification process and card purse dynamics.]
6. The void generates a reversal-class transaction which we take and place into the store and forward ('SAF') queue we maintain for this gateway connection.

The Terminal Reversal category actually tallies two separate transactions: the void (as described above) and reversals (occasions in which the store system didn't get a response from us within the alloted time period).  The former are customer-initiated decisions; the latter are system-generated actions.  We had a very clean day performance-wise, so I know that elevated number is almost purely the result of voids.

[For the record, Host Reversals - see other column - are something else entirely: it's when we (the payment switch) timeout the gateway or endpoint.]

This end-of-year customer behavior has quite a material impact on these metrics.  I went back and took a look at November statistics to get a worthy comparison.  For those 30 days, we - on average - tallied one (1) terminal reversal or void for every 5,174 transactions recorded with a standard deviation of 1,236.  For December 30th, the number was one per 1,573 transactions (a width of 2.9 standard deviations away from the November average).  For December 31st, it was down to 1,136 (3.2 SDs).  

Double Forward

With apologies to the Double Rainbow guy, I'm more excited about Federico's Double Forward. In a previous post, I laid out when and how to forward a transaction from one jPOS TransactionManager ('TM') to another.  Now, in our Pharmacy Adjudication and Local Editing implementation, we have a need for a Double Forward, i.e, one TM forwards to another, which in turn forwards to yet another.  Here, I lay out the details of Federico's nice little implementation of that requirement.

Over the past couple of years, we've refined our TM implementations such that a main TM - acting as a 'dispatcher' - pulls the requests from the inbound server queue, determines the transaction category, and the forwards to a second TM that acts as a worker.  In our financial implementation, for example, a Main TM parses and identifies the transaction and then forwards it to a Credit/FSA TM, Debit/EBT TM, or Stored Value TM (partial list) depending on the transaction type.  [See this post for some comments about how we implemented FSA in the TM.]

So, here are the significant parts of the Main TM here in our Pharmacy implementation:

In the place I've marked 'A', we forward some incoming transactions to an Rx Simulator we've built.  At point 'B', we do a case switch (based on the value the NCPDP Transaction Type) to either forward to our Local Editing TM or our Formatting TM.  For the record, the forwarding groups look like this:

Let's follow the Forwarding TM, because that's where the second forward takes place.  Here's the Formatting TM:

Notes:

A:  Here's our double forward - if we can't format the transaction for adjudication here in our environment (i.e., if we've not been provided formatting instructions), then we forward to the Stratus TM (more on that in a bit).

B:  We've put a concept in place called 'Transaction Regulator.'  As you can see here, results emanating from the regulator can change transaction flow.  [I'll have more to say on the Regulator in a subsequent post.]

C:  Here's a nice Select Endpoint.  You can see that we query either eRx (owned by Emdeon) or Relay Health to adjudicate the claim.  A neat addition here: this is a test TM that provides our customers the ability to reach the production eRx and Relay systems from their test and QA installations, thus the 'T2P' ("Test To Prod") option for both.

D:  Here's the forward to the Stratus TM.

E:  Note the 'Stats Collector' reference.  More on that in a subsequent post.

Finally, here's the bits of the Stratus TM of interest:

A:  We used the 'STRATUSP3.TXN' queue to forward.

B:  Pretty neat little thing here...if we are sending the transaction to the Stratus (the legacy environment), it has to look to that application like it came from the origination point...including sending it to the 'right port'...what I call our "port in, port out" implementation.  To that end, we've stuck the inbound port in the transaction context ('PORT') and then use that value here to branch to the right 'query host.'  [Note: the '@' variable values shown here are resolved at build time.]

C: Here we start the queries to the various Stratus destinations.  [We use the one-shot channel adapter here.]

Reaching a surprising barrier in our loyalty implementation

Here's an interesting problem to have: as part of the work we're doing with Market Basket Analysis ('MBA') transactions for our Loyalty Program efforts for a key client, we ran up against an interesting barrier this week.

The MBAs give us the line item detail of up to 40 products being purchased at the point-of-sale.  As part of our payment switch work, we act as the traffic cop and nexus point between 30,000 points-of-sale and our client's loyalty offer engine.  We fire a series of messages at the offer engine and - in response - get what we call 'triggered offers,' i.e, offers being tendered to the customer on the basis of what they're purchasing (as opposed to a 'targeted' offer, something tendered to the customer as a result of the marketing department assigning a 'loyalty message' to a group of cardholders).

We're doing over a million-and-a-half 'Loyalty Lookups' per day (which activate targeted offers, if applicable) and over half-a-million MBAs (which activate triggered offers, if applicable).  Most go smoothly.  But this past week we had a 40-item market basket which, in turn, generated an eye-popping 52 qualifying (i.e., triggered) offers.  [Quite a technical feat for us, too, as the offer engine we connect to sent in 56 or so separate messages to relay this information, which we needed to assemble and process as a 'single' reply.]

To transform these offers into meaningful messages for the point-of-sale, we have to go through a 'loyalty message lookup' followed by multiple passes at variable substitution and, if necessary, coupon serialization.  As a result, we can take something as pithy as this (as received from the offer engine):

  print-body: 'ID^01901|\n\n

...and turn it into hundreds of characters of text that is meaningful to a loyalty cardholder, the upshot being those foot-long (or, FEET-long) receipts you're getting at retailers these days!

Of the 52 qualifying offers mentioned above, that translated and congealed into over 24,000 characters of verbiage.  The barrier: the store system is only capable of supporting a 16,000-character message payload, a restriction we, in turn, enforced on our side by means of this jPOS FSD message:

The result: we only conveyed two-thirds of the offers (and probably caused an issue at the point-of-sale because the last of the offers was chopped off).  Now, we'll need to put our heads together with the offer engine and store systems team to come up with the best solution for all involved.  [I've got to say, I thought 16,000 was a "theoretical max" that was safely higher than anything we'd see in the real world.  But as is typical in these situations, reality quickly outpaced theory.]

Transaction categorization for Dashboard display

We spent some time working on our Dashboard display mechanics this week.  We're aiming to craft a nice one-page display for management - TPS and financial throughput.  The challenge is not to get too granular on that one page.  No one wants a page they need to scroll.  Management doesn't care about too much, for example, about the breakdown in EBT Cash vs. EBT Food transactions.  But, they would surely appreciate a one-screen glance at Debit/EBT as single TPS/$ tally, Credit as another tally, etc.

To that end, I offered to Dave and his development team that the transactions we process on financial payment switches arrange themselves into six nice categories: EV (sort of a grabbag of difficult to categorize applications); Check (or, more accurately, Eletronic Check Acceptance); Credit/FSA (also contains offline, or 'PIN-less,' Debit); Debit/EBT (birds of a feather); Loyalty; and Stored Value. 

The categorization came out looking [somewhat*] like this (column names are fields on our tranlog):

subclass  cardType cardBrand Description
--------  -------- --------- -----------
EV        AR       NULL      Authorized Returns
          ME       NULL      MethCheck

CK        CX       NULL      Electronic Check Acceptance

CR        CR       AX        Credit/AMEX
          CR       DS        Credit/Discover
          CR       JC        Credit/Department Store Card
          CR       MC        Credit/MasterCard
          CR       RC        Credit/Private Label
          CR       VI        Credit/Visa
          FS       AX        Flexible Spending/AMEX
          FS       HF        Flexible Spending/HealthFirst
          FS       MC        Flexible Spending/MasterCard
          FS       VI        Flexible Spending/Visa

DE        DB       NULL      Debit
          EC       NULL      EBT Cash
          EF       NULL      EBT Food

LO        CL       NULL      Customer Loyalty Program/Card
          EP       NULL      Employee Program/Card
          LB       NULL      Loyalty Bonus
          LC       NULL      Loyalty Coupon
          SC       NULL      Serialized Coupon

SV        CP       NULL      Incomm
          GF       NULL      Store-Branded Gift Card
          IN       NULL      Incomm
          PC       NULL      Verizon
          SV       NULL      Green Dot

* Small bit of editing in the table to protect some customer-specific information.

Auspicious week on the compliance front

It was an auspicious week for us on the compliance front.  After some significant work, we hit two milestone achievements...

First, our payment systems switching engine was awarded the VerAfied security mark by Veracode.  There's an official announcement of this achievement in the works.  I'll post that when it comes out. 

Second, we concluded a very productive stretch of work with K3DES, our Payment Application (PA) QSA, culminating in a successful outcome of certification against the PA-DSS v1.2 standard.

That's all I'm going to say here - my colleague Dave Bergert has a feature piece in the works on these subjects.  His credentials (CISSP, CISA, CPISM/A, and former QSA) give him a perspective on these matters that I can't begin to touch.

-------------------------

UPDATE - February 5, 2011:  See my new post entitled VERAFIED by Veracode

1 Customer, 1 Week, 42,572,884 Transactions

Related Readings:

More Than Just a Payment System

Life with ECA

An Update on Our Rx Adjudication Project Efforts

Rx Adjudication Volumes: Up, Up and Away

PCI Split

Virtuous Spiral (Take 1)

The Virtuous Spiral and PCI Split: Final Diagrams

------------------

Time for another quick look-in at our flagship switch location.  Last time I tallied up a week for review, we were nearing 32 million transactions over a seven-day stretch.  Now, we're a full one-third beyond that watermark.  Here's the tale of the tape:

[The xlsx file file can be retrieved here.]

As noted here, that's 2.2 billion transactions over a year.  We expect these numbers reflect the 'new reality' there for us, not just a surge.

A quick summary of the three implementations:

• Financial [PCI] - This one does traditional payment switch applications (Online Debit, Credit + Signature Debit, Stored Value, Flexible Spending, Health Care Savings and Electronic Check Acceptance)
  
• Financial [NOP] - The non-PCI split-off of the financial implementation does MethCheck, Authorized Returns, and a full-blown loyalty program (including market basket analysis and real-time offer engine integration)
  
• Rx Adjudication - A real-time engine we've built to help this customer's pharmacists make a "determination of the insurer's payment or financial responsibility."  [This Rx implementation will soon be expanded to include 'local edit' (a.k.a., 'pre-edit') functionality.]
  

Three factors have lead to the growth vs. my previous post in mid-August of this year:

• Rx Rollout is Complete - We're now at 100% of 'Phase 2' of our Rx rollout (see our three phase project described here).
  
• Traction on Loyalty - I've not spoken much about our Loyalty project efforts on these pages because we're standing coyly behind the hedges on that front; but we've spend a fair amount of the last year+ on that project.  Now, patience and learning has paid off and volume has sprouted some serious wings.
  
• Holiday! - We witnessed the annual rush to the stores over the days leading up to Halloween.
  

Rx Adjudication Volumes: Up, Up and Away

We're putting a foot firmly on the accelerator at our big Rx Adjudication implementation.  [See details on the three phase rollout of this project here.]  We've just put the wrap on our first working week in which every day topped the three-million transaction threshold:

About the Rx Adjudication project: We're drawing on all the practices and lessons learned in our financial implementations but face some new challenges.  The most prominent of those: dealing with the sheer size of the individual transaction requests and the corresponding requirement that we log (into a database) a full and faithful copy of all four message requests: the request from the Host (acting as a way station from the stores - this is our so-called 'HRQ' leg); the outbound request to the adjudicator ('REQ'); the response from the adjudicator ('RSP'); and the response to the Host ('HRS').

Ratcheting up the pressure even further: the 'HRQ' arrives in what we call 'fully expanded' format, i.e., it contains all possible fields in the NCPDP adjudication request.  That tally comes to 3,971 bytes in NCPDP 5.1 Format.  When our customer makes the transition to the NCPDP D.0, the bar goes up higher even still: a fully expanded D.0 HRQ weighs in at a whopping 10,595 bytes.  For those of you working in a purely financial world - one in which a credit or debit transaction arrives in a tight little package (typically less than a couple of hundred characters), the sheer size of adjudication messages (not to mention the attendent logging responsibilities) brings with it an entirely new set of challenges.

While our application held up very well under the strain of this weight, we were really putting the screws to the underlying database (MS SQL Server 2008 in this case) - a direct result of the HRQ, REQ, RSP, HRS logging.  It wasn't so much our inherent OLTP operations causing issues.  It was the all-important 'behind the scenes' DB work - the shadow copies, index re-orgs and culling (see here and here) - that was having trouble executing to completion while OLTP operations took on more and more volume.

From here, I'll paste in my release notes to tell you how we resolved this constraint:

-----------------

1. RX-69 – Add compression option to raw_message table

Each Phase 2 business transaction can log up to six raw_message entries (four for Phase 2 pass, two for Phase 1 pass). The ‘fully expanded,’ fixed mainframe request message (‘HRQ’) is particularly large (almost 4,000 bytes) and will grow dramatically with the introduction of the NCPDP D.0 standard (> 10,000 bytes). Dave Bergert of OLS suggested compressing these.  Here are his notes to [the client's DBA]:

I did a quick test on compressing the raw_message entries.  

Note these NCPDP messages have a TON of repeating characters like spaces and zeros, FS, GS, etc.

Length: 18199
Compressed Length: 2381

The process took approximately: 4 milliseconds

18K is much larger then what we expect - so this operation probably takes 2.5ms per Raw Message and saves 90% of the space.

At six raw messages per transaction, these steps will add less than 15 ms to the process. 

Then, [the client's] RX UI would just need to ‘unzip’ the blob to view the raw messages.

This change introduces three new columns to the raw_message table:

• [compressed] [bit] NULL – 0 = uncompressed, 1 = compressed
• [raw_size] [int] NULL – Records size of the uncompressed record
• [comp_size] [int] NULL – Records size of the compressed record (populated with 0 if ‘compressed’ = 0)

By comparing comp_size to raw_size, [the client] can keep metrics on the effect of compression on the various message legs.

2. RX-72 – Do a single insert for each tran_stats, raw_message entry near the end of a transaction

With Phase 2 occurring, we now log six messages to raw_message: HRQ, REQ, RSP and HRS on Phase 2 pass (new); and REQ, RSP on Phase 1 pass (existing). HRQ – in particular – is substantially larger than anything logged to date due to its 3971-byte fixed nature.

A note from [the client's DBA] summarizes his group's observations about the raw_message table:

From: [Grandmaster DBA]
Sent: Thursday, August 05, 2010 9:49 AM
To: OLS
Subject: OLS RX

I am seeing an update to the raw_message at the end of the transaction. In this particular transaction, it had 3 inserts to raw_message during the flow, but then also fired 3 updates at the end.

update raw_message set tranId= @P0 , date= @P1 , messageType= @P2 , messageData= @P3 , channel= @P4 , node= @P5 where id= @P6

Since raw_message is never updated, [the DBAs] ask if all DB-related raw_message work can be moved to end of the transaction. As things stand now, we create the Phase 2 raw_message objects at the beginning of the transaction, then pass the transaction over to our Phase 1 env. through the [legacy switch]. The Phase 2 DB transaction remains open throughout. [The DBAs] note that the result is a sustained locking effect on raw_message. The immediate fallout has been the inability to successfully purge records in the evening.

We are implementing [the DBAs'] suggestion as part of this release.

From: David Bergert
To: Andy, Rick

Regarding recent commits this evening - we will now only do a single insert for each tran_stats and raw_message entries near the end of a transaction, rather than the Tranlog method of creating the row, receiving the tranlog id, inserting whatever request data we have and updating the results on completion of the transaction. This change should address the DBA's ‘page-locking in MSSQL server’ concerns.

For further information, you can read Dave's background on his compression work here.

Debit vs. Credit Metrics

Here's a nice transaction summary that speaks to the key difference between PIN-ed debit (aka, online debit) and credit (a category that also includes offline debit, aka PIN-less debit, and Flexible Spending Accounts):
 

You can see the difference between the two categories: Debit clocks in pretty reliably at 300ms greater than Credit.  That's the Hardware Security Module ('HSM').  Every institution along the transaction path has to make a call to an HSM to either translate or validate the accompanying PIN.  I talk about this effect in more detail in the comments attached to this earlier post.