Andy Orrock | Payment Systems

Valentine's Day 2011

Our flagship acquirer central switch location had a massive volume day yesterday.  Since Valentine's Day fell on a Monday, we witnessed last-minute spouse/'significant other' gift buying dovetailing with the peak day of the week for pharmacy adjudication.  The result: over eight million transaction requests processed in the calendar day.  Here's the tale of the tape:

Financial Non-PCI  - 3,437,388
RX Adjudication    - 3,156,978
Financial PCI      - 1,484,299
                     ---------
Total              - 8,078,665

Of the Non-PCI total:

Market Baskets     - 1,262,848
Loyalty Lookups    - 2,097,035
AuthRtn, MethCheck -    77,505
                     ---------
Total              - 3,437,388

The most interesting part of these numbers for me personally is to compare them to those quoted in my Valentine's Day posts of 2008 and 2009.  Back then, I noted peak days of 1,168,700 and 1,213,311 respectively.  What's driven the sixfold (plus) growth since then?  First, we've worked with this client to create new programs.  Most notably, a new loyalty initiative over the past two years has led to us performing market basket analysis and doing loyalty card lookups.  As you can see, that's three million plus added to the total right there.

Second, our success at payment processing led to an invitation to displace a legacy Rx adjudication implementation.  That's another three million plus. 

It's a confluence of our increased experience, confidence placed in us and our client's creativity unleashed.  It's the virtuous spiral in action.

VERAFIED by Veracode

OLS.Switch (both the issuer and acquirer implementations) is now officially VERAFIED by Veracode.  Dave Bergert and his development team worked arduously to attain this status.  As explained by Veracode...

The VERAFIED security marks signify that a software provider has taken appropriate steps to remove vulnerabilities in their software or to comply with respected industry standards such as the OWASP Top 10 or the CWE/SANS Top 25 Most Dangerous Software Errors. A VERAFIED mark and Directory listing provides insight into the security quality of the software similar to the insights provided to their customers by independent organizations such as Standard and Poor's and Consumer Reports.

Here's a nice summary page of our software and the 'Assurance Level' bestowed upon us by Veracode -- AL5 (Very High) Mission critical for business/safety of life and limb on the line -- their highest designation.

Additionally, here are some video testimonials that Veracode elicited from us - Dave, OLS' CEO Terry Richards and yours truly talking about the challenges of being a payment systems software provider, what led us to Veracode and why the Verafied mark was important to us.

We're thrilled to be recognized by Veracode.  It was a pleasure to work with them.

Summary to-date of our loyalty work

I've not spoken a lot in this blog about Loyalty, but the reality of the matter is that if I look at how we've spent our time over the past two to three years, it has been apportioned into approximately 40% Loyalty, 40% Pharmacy Adjudication and 20% traditional Payment Systems (which tells that maybe I need to rename my blog!)

Over the past couple of years, OLS has practiced the art of on-boarding new, high-volume transaction services with diligence. This work has created a ‘virtuous circle’ for our customers: as we get better and more efficient at what we do, we unleash their creativity. This, in turn, leads them to envision and request ever more inventive value-added services that – while anchored in the practices honed in our core payment backbone – allow us to stake out and build new areas of competencies.

Foremost of these efforts over the past couple of years has been our work on customer loyalty programs. These programs have allowed our key clients to cement their relationship with their best customers, providing them with special offers, encouraging repeat visits and increasing the size of their market basket.  At our flagship loyalty implementation, we help drive this behavior for 30 million customers on file.  Whether these customers enter one of more than 4,000 store locations coast-to-coast or visit the web, our software helps control and shape the user experience.

But, what makes up this loyalty implementation?  Last month, Dave was doing some follow-up to an RFP response and had a need to provide a summary of our efforts.  I created a two-page brief for him, which I've posted here.  In that document, I broke down our implementation into the following components:

• Loyalty Card Management
• Loyalty Coupon Management
• Coupon Serialization Manager
• Promotion Code Management
• Real-time Offer Engine Integration
• Point-of-sale integration
• Web Integration (via a RESTful interface)
  

We've been aggressively expanding our capabilities in these arenas over the past 18 months.  Of all the work we do, it's loyalty where we've really got the foot on the accelerator in terms of creating and introducing new services.  As testament to that statement, I make note that the document I've referenced here - written less than three weeks ago - is already materially out of date.  We've just delivered electronic coupon ('Ecoupon') functionality for client QA.  Moreover, Dave and his development team are staking out their moves on mobile as the next logical channel for loyalty integration.

Multiple nodes for the win!

A few weeks back we had the 'opportunity' to exploit the benefits of our multiple node payment systems deployment strategy.  In general, we manage our release process very carefully.  We preach and practice small, iterative releases.  That approach reduces risk, both by controlling the rate of change and by not letting intertwined initiaitves pile up in the test and QA environments. 

Despite this diligence, there are situations that come along that force an emergency middle-of-the-day install of corrective functionality.  Thankfully, these situations are rare.  But, when they do pop their head, it's good to know we have an approach that minimizes impact.  In this particular instance, our customer has an OLS.Switch implementation with two replicated application nodes fronted by a Cisco Content Sensitive Switch ('CSS' or, more commonly, a 'load balancer').  At 2:00 PM that weekday, we implemented the updated release:

1. Stop the service on App Server 1 (all in-flight transactions complete before the service stops).
2. [CSS routes all transactions to App Server 2.]
3. Place the updated release into position on App Server 1.
4. Start the service on App Server 1.
5. [CSS re-balances all transactions between App Server 1 and 2.]
6. Repeat steps 1 through 4 for App Server 2.

That process took about five minutes start to finish.

In the days of legacy payment switches, undertaking an exercise like this mid-day would be nearly unthinkable.  It speaks to the unspoken truth about legacy OLTP implementations: that the purveyors tout their 'five nines' uptime, but speak only of unplanned outages.  The dirty little secret is the fear and anxiety related to planned outages.  With hoary, creaky applications that take 45 minutes to cycle and boxes teetering on the edge of their CPU capacity, a mid-day application cycle is a death-defying act, if not a bet-your-job proposition.

Observations about FDR and Amazon 2010 peaks

I really liked this recent post by Alejandro about jPOS performance -- not just because of the gracious references to me and Ron, but because it slyly speaks to a common discussion point between me and Alejandro: the hyperinflation of TPS expectations and projections (the result of business plans that say something like "if we only get 1% of the world's volume which should be easy, then that translates to...") vs. the reality of what some of the world's largest established processors do right now.

It's interesting to see in that post that FDR (FDR!) hit 988 TPS on its Black Friday peak.  I realize that the FDR universe is split up into various pieces and that that article is probably about what FDR calls its 'Central' platform only, but still: one of the major gateways in the US is clocking a peak < 1,000 TPS.  On the other hand, both Alejandro and I have met people plan in hand, 0 TPS in the books, 2,500 TPS in head.  [Let's focus on you getting Transaction 001 in the house first, guy.]

In that same vein - here's the world's largest e-tailer announcing its peak Cyber Monday results: 158 items per second (and far fewer checkouts when you think in terms of holiday-sized market baskets).

Transaction counts at 2010 year-end

Now that 2010 has drawn to a close, I was interested in where we sit in terms of 'transactions processed to date' at our largest acquirer-side payment switch implementation.  Here's the tale of the tape:

Rx:                    576,259,057 [Adjudication]
Financial [Non-PCI]:   395,830,634 [Mostly Loyalty]
Financial [PCI]:     1,293,382,153 [Debit, Credit, ECA, SV]
                     -------------
Total:               2,265,471,844

Even though the Financial PCI implementation has the biggest number, it's only a reflection of it being the first of our three implementations there.  The other two systems are growing at far greater rates.

Related Posts:

PCI Split

An update on our Rx adjudication project efforts

Live with ECA

Implementing MethCheck

 

Interesting EOY Reversal Dynamics

At one of our acquirer-side payment switch implementations, the terminal-generated reversal/void count is exhibiting its annual end-of-year spike:

What's going on here?  This phenomenon is the result of people rushing to this nation-wide retail chain looking to fully use their Flexible Spending or Health Savings Accounts ('FSA/HSA') prior to those benefits expiring at end of year.  Our implementation supports partial approvals of these transactions.  What's happening in these situations is:

1. We format and send the transaction to our client's FSA/HSA gateway (here, the excellent FDR North network).  As part of this transaction, we turn on an indicator asking FDR to inform the card issuer that we're capable of supporting partial approvals.
2. We get an approval back along with the original amount we requested and the amount the issuer approved.
3. We inform our client's store system of the approval amount.
4. In cases where the approved amount is a partial, the store system instructs the associate to ask for another form of payment to cover the remaining amount.
5. At this point, most people provide the funds to complete the sale from another source.  But, in many cases, the customer elects to void the sale or reduce the size of their market basket.  [This is understandable behavior: many people aren't aware of the exact balance of their card; and HSA transactions are complicated by the item qualification process and card purse dynamics.]
6. The void generates a reversal-class transaction which we take and place into the store and forward ('SAF') queue we maintain for this gateway connection.

The Terminal Reversal category actually tallies two separate transactions: the void (as described above) and reversals (occasions in which the store system didn't get a response from us within the alloted time period).  The former are customer-initiated decisions; the latter are system-generated actions.  We had a very clean day performance-wise, so I know that elevated number is almost purely the result of voids.

[For the record, Host Reversals - see other column - are something else entirely: it's when we (the payment switch) timeout the gateway or endpoint.]

This end-of-year customer behavior has quite a material impact on these metrics.  I went back and took a look at November statistics to get a worthy comparison.  For those 30 days, we - on average - tallied one (1) terminal reversal or void for every 5,174 transactions recorded with a standard deviation of 1,236.  For December 30th, the number was one per 1,573 transactions (a width of 2.9 standard deviations away from the November average).  For December 31st, it was down to 1,136 (3.2 SDs).  

Double Forward

With apologies to the Double Rainbow guy, I'm more excited about Federico's Double Forward. In a previous post, I laid out when and how to forward a transaction from one jPOS TransactionManager ('TM') to another.  Now, in our Pharmacy Adjudication and Local Editing implementation, we have a need for a Double Forward, i.e, one TM forwards to another, which in turn forwards to yet another.  Here, I lay out the details of Federico's nice little implementation of that requirement.

Over the past couple of years, we've refined our TM implementations such that a main TM - acting as a 'dispatcher' - pulls the requests from the inbound server queue, determines the transaction category, and the forwards to a second TM that acts as a worker.  In our financial implementation, for example, a Main TM parses and identifies the transaction and then forwards it to a Credit/FSA TM, Debit/EBT TM, or Stored Value TM (partial list) depending on the transaction type.  [See this post for some comments about how we implemented FSA in the TM.]

So, here are the significant parts of the Main TM here in our Pharmacy implementation:

In the place I've marked 'A', we forward some incoming transactions to an Rx Simulator we've built.  At point 'B', we do a case switch (based on the value the NCPDP Transaction Type) to either forward to our Local Editing TM or our Formatting TM.  For the record, the forwarding groups look like this:

Let's follow the Forwarding TM, because that's where the second forward takes place.  Here's the Formatting TM:

Notes:

A:  Here's our double forward - if we can't format the transaction for adjudication here in our environment (i.e., if we've not been provided formatting instructions), then we forward to the Stratus TM (more on that in a bit).

B:  We've put a concept in place called 'Transaction Regulator.'  As you can see here, results emanating from the regulator can change transaction flow.  [I'll have more to say on the Regulator in a subsequent post.]

C:  Here's a nice Select Endpoint.  You can see that we query either eRx (owned by Emdeon) or Relay Health to adjudicate the claim.  A neat addition here: this is a test TM that provides our customers the ability to reach the production eRx and Relay systems from their test and QA installations, thus the 'T2P' ("Test To Prod") option for both.

D:  Here's the forward to the Stratus TM.

E:  Note the 'Stats Collector' reference.  More on that in a subsequent post.

Finally, here's the bits of the Stratus TM of interest:

A:  We used the 'STRATUSP3.TXN' queue to forward.

B:  Pretty neat little thing here...if we are sending the transaction to the Stratus (the legacy environment), it has to look to that application like it came from the origination point...including sending it to the 'right port'...what I call our "port in, port out" implementation.  To that end, we've stuck the inbound port in the transaction context ('PORT') and then use that value here to branch to the right 'query host.'  [Note: the '@' variable values shown here are resolved at build time.]

C: Here we start the queries to the various Stratus destinations.  [We use the one-shot channel adapter here.]

Reaching a surprising barrier in our loyalty implementation

Here's an interesting problem to have: as part of the work we're doing with Market Basket Analysis ('MBA') transactions for our Loyalty Program efforts for a key client, we ran up against an interesting barrier this week.

The MBAs give us the line item detail of up to 40 products being purchased at the point-of-sale.  As part of our payment switch work, we act as the traffic cop and nexus point between 30,000 points-of-sale and our client's loyalty offer engine.  We fire a series of messages at the offer engine and - in response - get what we call 'triggered offers,' i.e, offers being tendered to the customer on the basis of what they're purchasing (as opposed to a 'targeted' offer, something tendered to the customer as a result of the marketing department assigning a 'loyalty message' to a group of cardholders).

We're doing over a million-and-a-half 'Loyalty Lookups' per day (which activate targeted offers, if applicable) and over half-a-million MBAs (which activate triggered offers, if applicable).  Most go smoothly.  But this past week we had a 40-item market basket which, in turn, generated an eye-popping 52 qualifying (i.e., triggered) offers.  [Quite a technical feat for us, too, as the offer engine we connect to sent in 56 or so separate messages to relay this information, which we needed to assemble and process as a 'single' reply.]

To transform these offers into meaningful messages for the point-of-sale, we have to go through a 'loyalty message lookup' followed by multiple passes at variable substitution and, if necessary, coupon serialization.  As a result, we can take something as pithy as this (as received from the offer engine):

  print-body: 'ID^01901|\n\n

...and turn it into hundreds of characters of text that is meaningful to a loyalty cardholder, the upshot being those foot-long (or, FEET-long) receipts you're getting at retailers these days!

Of the 52 qualifying offers mentioned above, that translated and congealed into over 24,000 characters of verbiage.  The barrier: the store system is only capable of supporting a 16,000-character message payload, a restriction we, in turn, enforced on our side by means of this jPOS FSD message:

The result: we only conveyed two-thirds of the offers (and probably caused an issue at the point-of-sale because the last of the offers was chopped off).  Now, we'll need to put our heads together with the offer engine and store systems team to come up with the best solution for all involved.  [I've got to say, I thought 16,000 was a "theoretical max" that was safely higher than anything we'd see in the real world.  But as is typical in these situations, reality quickly outpaced theory.]

Transaction categorization for Dashboard display

We spent some time working on our Dashboard display mechanics this week.  We're aiming to craft a nice one-page display for management - TPS and financial throughput.  The challenge is not to get too granular on that one page.  No one wants a page they need to scroll.  Management doesn't care about too much, for example, about the breakdown in EBT Cash vs. EBT Food transactions.  But, they would surely appreciate a one-screen glance at Debit/EBT as single TPS/$ tally, Credit as another tally, etc.

To that end, I offered to Dave and his development team that the transactions we process on financial payment switches arrange themselves into six nice categories: EV (sort of a grabbag of difficult to categorize applications); Check (or, more accurately, Eletronic Check Acceptance); Credit/FSA (also contains offline, or 'PIN-less,' Debit); Debit/EBT (birds of a feather); Loyalty; and Stored Value. 

The categorization came out looking [somewhat*] like this (column names are fields on our tranlog):

subclass  cardType cardBrand Description
--------  -------- --------- -----------
EV        AR       NULL      Authorized Returns
          ME       NULL      MethCheck

CK        CX       NULL      Electronic Check Acceptance

CR        CR       AX        Credit/AMEX
          CR       DS        Credit/Discover
          CR       JC        Credit/Department Store Card
          CR       MC        Credit/MasterCard
          CR       RC        Credit/Private Label
          CR       VI        Credit/Visa
          FS       AX        Flexible Spending/AMEX
          FS       HF        Flexible Spending/HealthFirst
          FS       MC        Flexible Spending/MasterCard
          FS       VI        Flexible Spending/Visa

DE        DB       NULL      Debit
          EC       NULL      EBT Cash
          EF       NULL      EBT Food

LO        CL       NULL      Customer Loyalty Program/Card
          EP       NULL      Employee Program/Card
          LB       NULL      Loyalty Bonus
          LC       NULL      Loyalty Coupon
          SC       NULL      Serialized Coupon

SV        CP       NULL      Incomm
          GF       NULL      Store-Branded Gift Card
          IN       NULL      Incomm
          PC       NULL      Verizon
          SV       NULL      Green Dot

* Small bit of editing in the table to protect some customer-specific information.