Energy-market opening on SAP IS-U and CRM

From July 2007, the French residential energy market opened to competition: Gaz de France could sell electricity and EDF could sell gas. GDF, with 11 million customers in 2007, needed an information system for its commercial offers, contracts and billing, including new dual-fuel (gas plus electricity) offers. The program implemented SAP CRM for customer relationship management, SAP IS-U for billing and sales administration, and BW for steering indicators. More than 200 consultants worked on it, one of Accenture's largest projects at the time. Joined in December 2006 as an ABAP programmer in a team of 20 developers, part of them offshore in Mauritius, for the release that added everything needed to sell electricity.

The challenge

Two releases were already live and covered gas sales and billing. Version 3 had to bring everything needed to sell electricity, plus many enhancements and optimizations, and it had to be ready for the market opening in July 2007. The pressure was high.

The development team was split between production maintenance of the live releases and evolution (new features). I started my career on the evolution side. Each developer could work on IS-U or CRM; I worked on both, with a little more emphasis on CRM. Standard SAP did not always meet the client's many requirements, so the project combined a lot of configuration with custom ABAP.

Project organization

The project followed a V-cycle:

The build sat at the bottom of the V. Analysts, usually more experienced consultants, turned each functional need into a detailed technical specification, and I turned those specifications into algorithms and ABAP programs. In the weekly team meeting I reported progress, shared technical findings, flagged blockers early and committed to delivery dates.

• Unit tests: technical robustness. The program ends cleanly in every situation, handles bad input and gives explicit error messages. Then I handed over to the analyst for code review.
• Integration tests: test teams ran functional test books and logged defects in the defect-tracking tool. I used it constantly to read defects, fix programs and comment, and I worked directly with functional analysts, sometimes showing that a reported defect was in fact correct behavior.
• UAT: users ran business-scenario test books, and I reproduced and fixed new defects.

Test environments held real contracts, addresses and phone numbers. I was responsible for never exploiting that data or letting it leave the client's environments.

Technical deep dive: a reusable file-to-IDoc inbound pattern

The problem. IDocs are designed mainly for SAP-to-SAP exchange. Flat files, XML and web services were alternatives, but none of them had integration monitoring built into CRM and IS-U. Custom flat-file loaders, whether SQL or standard function calls, have to generate their own logs, which is tedious, inconsistent and spread across tools. The client wanted to track every integration: classify and search them, see their status at a glance, get explicit error messages and replay failures.

The solution. I used SAP IDocs even for non-SAP-to-SAP inbound flows, which avoided heavy custom development. Files landed on an SAP-side FTP area. Custom ABAP read them with file commands and loaded the data into IDoc structures, which served as the staging buffer.

The example I built was the electricity contract-modification interface in CRM. An upstream system dropped XML files into a dedicated server directory with contract updates, such as terminations, and meter-reading data. Standard SAP can post contract and meter IDocs with no development by calling the standard function modules linked to those IDoc types. Custom fields and tables also had to be updated, which I did with user exits in the standard code.

Processing steps:

1. Read the file and run an integrity check.
2. On error, write the log and error message and end. Otherwise continue.
3. Map the file lines to IDoc segments.
4. Create the IDocs.
5. Post the IDocs, which updates the contract data and the meters.
6. IDocs that fail to post are set to error status and feed a functional rejects table.
7. Produce the execution report and end.

Logs and database updates are written along the way.

Why it worked. IDocs created this way get SAP's built-in monitoring with no extra development: integrations per day, status, and reprocessing of errors. Production support became much easier, and users could check that their files had arrived in SAP and see what went wrong. The project's other inbound interfaces followed the same architecture, which gave uniform, reliable handling of inbound messages.

Technical deep dive: performance at 11 million customers

Constraints. The client set performance standards for every custom extraction:

• The nightly batch window had to finish before the business day, and before the batches that depended on it.
• Documents had to save in acceptable time.
• Customer and invoice searches had to be fast. Hotline agents taking customer calls needed customer and contract data almost instantly.
• Reports over all customers and invoices had to finish on time.
• Tables could grow to tens or hundreds of millions of rows, exposing slowness that did not appear in tests.

Process.

1. Performance defects from integration testing and UAT were assigned to me.
2. I reproduced them in a dedicated performance environment holding millions of rows, copied from production where possible or transformed to mask sensitive data such as bank details.
3. I measured with SAP's runtime-analysis trace, which records every statement and its duration, then fixed the code and redelivered it for re-test, checking both correctness and speed.

The landscape was the classic three-tier R/3 architecture: SAP GUI, several ABAP application servers sharing the load, and one Oracle database. Bottlenecks could sit in any tier.

Code fixes:

• Read internal tables with binary search instead of linear reads. Binary search only works on the sort key, so I often copied and sorted the table first.
• Removed nested loops, pulling the work out of the main loop into simple table reads, and sometimes redesigned the algorithm.
• Kept expensive operations such as SORT out of loops.
• Dropped useless processing, fetched only the relevant data and exited early.

SQL fixes:

• Used the SQL trace to see join order and index use, and added selection criteria so that an existing index was used.
• Treated a new index as a last resort. An index is a physical sorted copy: it takes space, slows inserts and updates, and can hurt overall performance.
• Limited the number of queries and read in bulk, then processed in application-server memory. Disk is roughly a thousand times slower than RAM, and the application servers work in parallel.
• Used table buffering only for configuration tables that do not change.
• Passed hardware and configuration issues to the Basis team with my analysis and recommendation. Database work also drew on Oracle knowledge of indexes, statistics and hints.

What I learned

• Rigor pays. Fewer integration-test defects was the main thing my managers judged me on.
• Strict adherence to development standards and naming rules, plus reviewing colleagues' code, gave me disciplined ABAP habits. Those habits later let me train new developers.
• I reused the file-to-IDoc pattern on later projects at Allianz, Total and L'Oréal.
• Performance is a large part of client satisfaction, and it is mostly won in the algorithm and the SQL, not in the hardware.