H-1B Visa Sponsors: Surprise! You're Being Audited - CIO.com - Business Technology Leadership

H-1B Visa Sponsors: Surprise! You're Being Audited - CIO.com - Business Technology Leadership

Posted using ShareThis

Host Integration Solutions

For host-integration, there are multiple products available in the market, which are based on different integration pattern (screen-scrapping, proxy, peer-to-peer etc.) for application & data integration. Saying that, let’s review the types of products available in the market at present:

1. Upgrading CICS TS: In CICS Transaction Server 3.1 & 3.2, IBM has given facility to expose CICS transactions as web-service. However, I think some amount of configuration or coding may be required. I have not done a detail-study on this.
   
   
   1. Advantage: No additional tool is required and it is well supported by IBM.
   2. Disadvantage: The enterprise may have an older version of transaction servers. This approach would require upgrading all transaction servers to the latest version (which may be costly & time-taking exercise).

1. Upgrading CICS TS + SOLA: The research team of Meryll Lynch developed an add-on tool on top of CICS TS 3.1, which is now sold to SOA Software. This tool provides complements CICS TS 3.1, and provides support to other standards of WS-* standards. Also, it reduces the development and deployment cycle. In production, this tool is installed on Mainframe with a small footprint.
   
   
   1. Links: http://www.soa.com/index.php/section/company_press_detail/soa_software_acquires_mainframe_web_services_product_from_merrill_lynch/
   2. Link-SOLA Overview: http://www.soa.com/index.php/section/sola/sola_mainframe_web_services/
   3. Advantage: Get all benefits of CICS TS 3.1 with additional facilities provided by the tool. Reduction in development & administration cycle may justify for SOLA tool.
   4. Disadvantage: If the enterprise has an older version of transaction, this approach would require upgrade of transaction server to version 3.1.

1. Using tools deployed on Mainframe: There are products like IBM's HATS, Ivory, HostBridge, which are installed on Mainframe and exposes the CICS transactions, 3270 programs, IMS transactions as web-services over http or MQ. This requires a small footprint on mainframe. No other tool is required.
   
   
   1. Links: http://www.hostbridge.com/
   2. Advantage: No change in existing code is required. All existing CICS transactions are exposed as web-service.
   3. Disadvantage: The product installed on Mainframe would consume additional Mainframe resources in terms of memory and CPU cycle. If current usage of mainframe resources is near its peak, additional budget would be required to upgrade the mainframe resources.

1. Using Middleware (adapters) like (DataDirect’s Shadow, LegaSuite, Microsoft's Host-Integration Server): There are few middleware products available, which sit between mainframe and the client application. It intercepts the SOAP requests, translates into COMMAREA message and invokes CICS transaction over TCP/IP or SNP.
   
   
   1. Link: DataDirect- http://www.datadirect.com/products/mainframe-integration/shadow-rte/index.ssp
   2. Link: LegaSuite- http://www.seagullsoftware.com/products/legasuite.html
   3. Link: Microsoft’s HIS- http://www.microsoft.com/hiserver/evaluation/overview/default.mspx
   4. Advantage: No change in existing application.
   5. Disadvantage: One more layer (middleware) is introduced in the system. This layer has its own landscape in terms of application and hardware.

Other than that, there are few other products available based on screen-scrapping or to provide access to data-sets/IMS/DB2 database on mainframe. These solutions are not complete solutions to access CICS & IMS transactions.

In my opinion, all the above options should be evaluated for the current and long-term vision. Evaluation would require participation from multiple vendors and creating a comparison-matrix. As of now, there is no clear-cut mandate on which one is the best product or the best integration pattern. The decision varies from one company to another company on the basis of requirements of volume, transaction etc. as well as the size of the company. A small size company may go for different solution/product than a huge enterprise depending upon various factors like budget, extend of reuse of mainframe resources, future landscape etc.

Java RTS 2.0 Update1 Released

Java RTS 2.0 Update-1 (90 days evaluation copy) is available for download. (Link: Download Java RTS)

The version available on Sun's site is for Solaris (both x86 & SPARC) only. IBM offers another version which runs on Linux with open-source real-time Linux extensions. There is no version available for Windows (with third party real-time extension). This update release provides support for Solaris 10 u4 and J2SE 5.0 u13 libraries as well as various security enhancements.

Java RTS features are delivered via a separate real-time enabled version of the Java VM. This VM can process both traditional Java SE components and Java RTS components. With Java RTS, real-time components and non-real-time components can coexist and share data on a single system.

This release is compliant with Real-Time Specification for Java (RTSJ). In fact, it is the first commercial implementation of RTSJ.

Here is more information on Java RTS (Taken from Sun's Document - as it is)

Here are the important features and benefits offered by the RTSJ and Sun's Java RTS implementation:

• New Real-Time Threads, Scheduling, and Synchronization

  The RTSJ introduces the concept of two new threads: real-time threads and no-heap real-time threads (a thread which cannot be interrupted by garbage collection). These threads offer more precise scheduling than with standard Java threads. They have 28 levels of priority and unlike standard Java, their priority is strictly enforced.

  Real-time threads are synchronized and are not subject to so-called priority inversion situations, where a lower priority thread has a block on a resource needed by a higher priority thread and thus prevents the higher priority thread from running. Rigorous testing with partners has shown that Java RTS completely avoids any priority inversions - which is crucial for mission-critical applications.

• New Memory Management Schemes

  The RTSJ defines two new types of memory areas that allow real-time applications to avoid unpredictable delays commonly caused by traditional garbage collectors:

  1. Immortal memory holds objects without destroying them except when the program ends. This means that objects created in immortal memory must be carefully allocated and managed.
  2. Scoped memory is used only while a process works within a particular section, or scope, of the program such as in a method. Objects are automatically destroyed when the process leaves the scope. This is a useful feature akin to garbage collection in that discrete creation and deletion is not required as in the immortal memory case - but the process must be sure to exit the scope to ensure memory is reaped.

  Neither immortal nor scoped memories are garbage collected, so using them avoids problems of GC interference.

• Asynchronous Events Handling & Asynchronous Transfer of Control

  The RTSJ provides two mechanisms for asynchronous communication: asynchronous event handling, and asynchronous transfer of control.

  Asynchronous event handlers deal with external events (known as "happenings") which can occur outside the JVM. The RTSJ is unique in that it allows developers to schedule the response to asynchronous events in order to avoid disrupting the temporal integrity of the rest of the real-time application.

  Asynchronous Transfer of Control (ATC) provides a carefully controlled way for one thread to interrupt another thread in a safe manner.

• Time & Timers

  The RTSJ specifies several ways to specify high-resolution (nanosecond accuracy) time including absolute time and relative time.

• Direct Access to Physical Memory

  While still maintaining security protections, the RTSJ allows direct access to physical memory. This means that device drivers can be created and written entirely in Java. Previously, Java applications had to link to native code to communicate directly with the hardware.

Java RTS Minimum Recommended System Requirements

• Dual core or dual CPU sytem with 512 MB
• Solaris 10 u3 SPARC or x86

What is a Real Time System (RTS)?

Real Time system does not mean a "fast" system. It means the system should respond to a worldly-event reliably and predictably. For example:

1. When an elevator reaches a floor, an event is generated. In response to this event, the system should stop the elevator immediately. If the system responds to this event late, the elevator would stop halfway. In this case, the timing is critical. If the system is very fast, but does not confirm to timing-contracts, it would not serve the purpose.
2. Another example is 'Anti-braking System (ABS)' in a car. When a wheel is locked, the system must respond to it by releasing it in timely manner. Any delay may have dangerous outcome.

A system is said to be real-time if the total correctness of an operation depends not only upon its logical correctness, but also upon the time in which it is performed. One of the definition (I found on Internet) is:

"A real-time system is one in which the correctness of the computations not only depends upon the logical correctness of the computation but also upon the time at which the result is produced. If the timing constraints of the system are not met, system failure is said to have occurred."

Most of the real-time systems are low level applications to control the physical hardware.

Hard and Soft Real Time System

In real time system, the time constraint is the one of the most important factor. In some applications, the time constraint is not flexible. Any minor deviation from the time constraint is considered as failure. This type of system is called Hard Real Time System. For example, a car engine control system is a hard real-time system because a delayed signal may cause engine failure or damage. Other examples of hard real-time embedded systems include medical systems such as heart pacemakers and industrial process controllers.

There are other scenarios where some latency is acceptable, which are called Soft Real Time System. Example: the software that maintains and updates the flight plans for commercial airliners. These can operate to a latency of seconds. Live audio-video systems are also usually soft real-time; violation of constraints results in degraded quality, but the system can continue to operate.

Components to implement a Real Time System

1. Operating System: The operating should support real time processing. It provides all the facilities to build a real-time application. It is a multi-tasking operating system and guarantees that the deadlines are met.
   
   VxWorks, QNX are few examples of operating system. Windows XP is not a real-time operating system. Through, there are few third party plug-ins, which are plugged into Windows XP to provide real-time facilities.
   
   If an application is developed & deployed on a real time operating system, it does not mean that the application is real-time. The application must be designed in a proper way to provide real-time behavior. It is similar to saying that everybody cannot drive the car. You need good training too for driving the car.
   
2. Application Framework: To create a new real-time system, the application development framework or environment should provide real-time facility too. For example: Java is not a real-time development environment. It has garbage collector, which can be initiated anytime to claim the free memory. When Garbage collection runs, it takes a lot of resources and other threads slow down. This delays in processing the event by other threads. This is one example where Java fails as real-time application development framework. Same is true for .NET too.
   
   Usually, C & C++ languages are usually used to develop real-time applications.
   
3. Application Design: The design of application should be such that it provides real-time functionality. If resources are available and design is not correct, you can't have a real-time solution.

Which persistence framework to use (JPA, JDO, Hibernate, Toplink) ?

With multiple persistence technologies in Java world, it was always difficult to evaluate all the existing frameworks and choose the best (& mature) one. Sun came up with unsuccessful technologies like entity beans and JDO. BEA created its own framework JDO based Kodo framework. Oracle's Toplink is another mature framework, which is in use for last several years. Open source communities create great persistence frameworks (Hibernate, iBATIS), which are being used heavily.

Java is an Object Oriented programming language, and it  was an unanimous desire to have a good object-relational mapping (OR Mapping) based persistence framework. In last few years, it has been established that Hibernate & Toplink are winner in this race. These are now considered as de-facto standard for persistence-framework. For last four years, I've seen Hibernate being used on all enterprise projects (except one, which was on iBATIS).

To compete with Hibernate and Toplink, Sun came up with OR-Mapping based framework "JDO". However, it failed to create an impression on java-community. BEA created its product Kodo based on JDO specification. Gavin King has highlighted few shortcomings in JDO in his article (Link: Gavin's Article). Currently, JDO provides more facilities than JPA (see comparison). However, Sun has indicated that JPA is the standard specification for OR-mapping based persistence. Here is one of FAQ from Sun (Link: JPA FAQ):

Q: What will happen to other data persistence APIs now that the Java Persistence API is available?

A: The Java Persistence API is now the standard API for persistence and object/relational mapping for the Java EE platform. Earlier APIs of course will not go away, but we expect that they will become less interesting once this new standard API is available.

Finally, experts from all popular frameworks (Hibernate, Toplink etc.) sat together to come up with a specification to define common APIs, that would be implemented or supported by the existing popular frameworks. And, JPA 1.0 was born. Hibernate, Toplink and other products were enhanced to support JPA specification.

With JPA, it is possible not to use one common APIs for many persistence frameworks. If the application is using JPA with Hibernate, it would not require any change in the code to replace Hibernate with Toplink or any other JPA based framework. Apache is working on another JPA based framework named as OpenJPA.

BEA and IBM are shipping OpenJPA as JPA implementation in their application servers WebLogic and WebSphere respectively. Oracle  has Toplink in its application server.

Conclusion

JPA is not the standard specification for OR-Mapping based persistence framework. It has support from all the major vendors including IBM, BEA, Oracle, SUN etc. Hibernate and Toplink are enhanced to support JPA. These two frameworks are in use for last several years and are mature now. It is easy to find developers with experience with these two frameworks. IBM and BEA are shipping a new OpenJPA framework, which it not proven as a mature product so far. So, in my opinion, the new projects should use Hibernate or Toplink with JPA even if the application would be deployed on WebLogic or WebSphere Application Server. And, while building the application, caution should be taken not to use any proprietary API from Hibernate or Toplink to provide portability in future.

Future of SCA after Java EE 6 ?

JSR 316 has been approved to create Java EE (version 6) specification, which is targeted to ship in 2008. (Link: JSR 316)

The scope of Java EE 6 includes the SCA-defined facilities. As per the JSR, the scope of Java EE 6 related to SOA is:

• SOA Support
  The Java EE platform is already widely used for SOA applications. As more and more enterprises realize the benefits of a SOA architecture, requirements on the platform have increased, both in terms of functionality and interoperability. 
  
  The Service Component Architecture is defining facilities that could be used by composite applications in a SOA environment. This expert group will consider whether any of the SCA-defined facilities are appropriate for inclusion in this release of the Java EE platform.

IBM and BEA are working together to make SCA technology as an alternative or replacement to existing Java EE technology. SCA is being promoted as a new framework to create services and components. It provides IoC containers and framework to create composite components. However, the current specification does not provide all facilities provided by Java EE (transaction, JNDI, connection-pooling, object-pooling, JCA, web-container etc.). So, the current version of SCA cannot be a replacement of Java EE. It can be used to create new services only.

Java EE is a mature technology being used by almost all Java based system for last several years. The customers are comfortable with this technology. Introducing new technology has its own challenges, risks and cost. Until there is a profit in moving to the new technology, which would require new software license, hardware, money, time, and resources, a CEO or CTO of an enterprise would not take a decision in favor of it.

The new version of Java EE 6 specification includes the new facilities offered by SCA to create new services. IBM and BEA should be building these new facilities in their application servers. With this, SCA may not provide any new facility, and there may not be a business case for an enterprise to build new services/ solutions based on SCA. Lets wait and see what IBM & BEA are doing with SCA.

SCA is an IoC Container, Component Model & Assembly Specification

IBM and BEA are pushing a new technology 'SCA' to build services. It provides a framework to create loosely coupled composite-application, where dependencies of components are defined declaratively. The components can be written in different languages and on different platform. Lets take a close looks of it with respect to existing technologies currently available starting with overview.

Overview of SCA

SCA defines a component framework to create services.  The basic building block for SCA is a component implementation. One component can compose of other components. The dependency between components is defined in the assembly-descriptor, which provide loose-coupling among components. The dependent components can be local or remote and can be implemented in different technologies.

A component implementation is described by the following attributes:

1. Services: This describes the function this type of component provides
2. References: This describes the dependencies this type of component has in order to function
3. Properties: This defines configuration parameters that can controls how the program logic can behave. For example, what currency to use.
4. Intent policies: This describes assumptions on how the component will behave. There are two types of policies.
   • Implementation policy- Impact the behavior of an implementation. For example, transaction, monitor and logging
   • Interaction policy - defines how the components behave with one another. For example, security.

The key benefits of SCA are:

1. Loose Coupling: Components integrate with other components without needing to know how other components are implemented. Components can be easily invoked either synchronously or asynchronously.
2. Flexibility: Components can easily be replaced by other components. It also provides the flexibility to create solutions using existing services (bottom up) or model the business and develop the solution (top down).
3. Productivity: It enables you to focus on your business logic rather than be slowed with concerns about infrastructure. SCA simplifies development experience for all developers (including J2EE and Integration Developers), integrators and application deployers.

Now, lets see sample codes to understand how a SCA based service is created. (All codes are taken from Apache Tuscany SCA user-guide : http://incubator.apache.org/tuscany/sca-java-user-guide.html)

The service is defined as an interface:

1 public interface CalculatorService {
2     double add(double n1, double n2);
3     double subtract(double n1, double n2);
4     double multiply(double n1, double n2);
5     double divide(double n1, double n2);
6 }

The service is implemented as Java Class:

 1 public class CalculatorServiceImpl implements CalculatorService {
 2 
 3     private AddService addService;
 4     private SubtractService subtractService;
 5     private MultiplyService multiplyService;
 6     private DivideService divideService;
 7    
 8     @Reference
 9     public void setAddService(AddService addService) {
10         this.addService = addService;
11     }
12 
13     ...set methods for the other attributes would go here
14 
15     public double add(double n1, double n2) {
16         return addService.add(n1, n2);
17     }
18 
19     ...implementations of the other methods would go here
20 }

In the code above, note that 'setAddService' method has an annotation 'Reference'. It indicates that 'AddService' is an interface. Its implementation will be defined in the assembly-descriptor (a XML file) using dependency-injection pattern. This is an example where SCA is acting as IoC (Inversion of Control) engine like Spring-framework.

The reference component 'AddService' is defined as an Interface:

1 public interface AddService {
2 
3     double add(double n1, double n2);
4 }

The component is implemented as Java Class:

1 public class AddServiceImpl implements AddService {
2 
3     public double add(double n1, double n2) {
4         return n1 + n2;
5     }
6 }

So far, a service and one component are defined and implemented. The service-implementation has a dependency on the component (AddService). This is defined in the assembly-descriptor:

 1 <composite xmlns="http://www.osoa.org/xmlns/sca/1.0"
 2            name="Calculator">
 3 
 4      <service name="CalculatorService" promote="CalculatorServiceComponent/CalculatorService">
 5          <interface.java interface="calculator.CalculatorService"/>
 6          <binding.jsonrpc/>
 7      </service>    
 8 
 9     <component name="CalculatorServiceComponent">
10         <implementation.java class="calculator.CalculatorServiceImpl"/>
11         <reference name="addService" target="AddServiceComponent" />
12         <!-- references to SubtractComponent, MultiplyComponent and DivideComponent  -->
13     </component>
14 
15     <component name="AddServiceComponent">
16         <implementation.java class="calculator.AddServiceImpl"/>
17     </component>
18 
19     <!-- definitions of SubtractComponent, MultiplyComponent and DivideComponent -->
20 
21 </composite>

In the code above, the 'reference' tag defines the implementation of 'AddService' component. This provides loose-coupling between components. If the implementation of AddService component is changed, change in assembly-descriptor is required only. The assembly-descriptor defines the binding and policy information too.

Lets review different aspect of SCA.

SCA as an IoC Container (Similar to Spring)

SCA's component can be compared to Spring's bean. SCA components have properties and references. The references are other dependent components. The dependent components (references) & properties are populated using the setters or constructors following injection design pattern. This is similar to the bean-definition in Spring-framework. Auto-wiring feature of SCA 1.0 is similar to Spring's Auto-wire feature too.

Looking at SCA specification, it seems that SCA is not providing any additional feature, which is not available in Spring IoC. However, if SCA IoC is provided by the SCA-runtime, it will be available for all the application by default. This would not require to include any other IoC container in the project.

SCA as Java EE Container

SCA provides a framework to create java components similar to EJBs. The new components are implemented as POJOs, and its attributes (security, binding etc.) are defined using annotation or assembly descriptor. IBM and BEA are suggesting to implement entire business-layer using SCA framework. However, Java EE specification provides other standard guidelines too:

• Transaction
• Thread Management
• Security
• Web Service Stype (Wrapped, Unwrapped, JAXB Serialization etc.)
• Object Pooling
• JCA
• Connection Pooling

Currently, SCA specification does not address above features in its version 1.0 specification. For another couple of years, it is certain that Java EE stack will be used by the projects. Until SCA matures and provides all the functionality available in Java EE, it is not likely that the customers would decide to move to this new technology. Lets wait and see how the industry takes its own course.

Assembly Descriptor

SCA defines an assembly descriptor to define the services, references and properties. SCA supports to use components created in different languages and platform to be used as references. This will help in creating a new service using the existing components written in other languages. But, the existing components cannot be used as SCA component as it is. It would require to create a SCA wrapper following SCA specification. In real world, there are very few cases where the client uses different technologies in one domain. The client may have multiple technologies in different domain. In that case, the integration should be done using orchestration (BPEL). Even SCA supports composition of components in a single domain.

Conclusion

SCA is an interesting technology and big product companies (IBM, BEA etc.) are behind it. However, this new technology is being offered as a replacement to a well established & mature Java EE technology. Very likely, the next release of Java EE (6) would include all the new features provided by SCA. SCA does not provide a roadmap whether it can be used as business-layer of web-application. Almost all enterprises have web-applications, and they want to have minimum set of technologies. Current Java EE technology provides a good framework to create new services. Bringing new technology to create new services or migration effort to transform current services (business layer) to new technology has its own challenges and risks.

Since big companies are behind SCA, it cannot be written off easily. They can try to force the existing user-base (like Microsoft) to use the new technology. But, it would not be an easy task for them. For next couple of years, it is not likely that the new projects would use SCA based system until it establishes itself as a mature technology similar to Java EE. We'll watch closely how it evolves.

JCP approved JPA 2.0 (JSR-317)

JCP (Java Community Process) has approved next version of Java Persistence API (JSR-317). Here is the link: JSR-317 (JPA 2.0).

The scope of JPA 2.0 is :

1. Expanded object/relational mapping functionality, including greater flexibility in combining existing mapping options, support for collections of embedded objects, multiple levels of embedded objects, ordered lists, combinations of access types, etc.
2. Additions to the Java Persistence query language
3. An API for "criteria" queries
4. Standardization of sets of "hints" for query configuration and for entity manager configuration
5. Standardization of additional metadata to support DDL generation and "Java2DB" mapping
6. Expanded pluggability contracts to support efficient passivation and replication of extended persistence contexts in Java EE environments
7. Standardization of additional contracts for entity detachment and merge, and persistence context management
   Support for validation

Its good to know that JPA 2.0 would include criteria API. I've been using Hibernate for last couple of years and have always liked criteria-API.

Almost all leading companies (including IBM, BEA, Oracle, Red Hat Sun etc.) have voted 'Yes' for this specification. Apache has voted 'No' on the ground that this spec lead - Sun - is in violation of the JSPA http://www.apache.org/jcp/sunopenletter.html and therefore shouldn't be allowed to start another JSR until the above matter is resolved. (Link: Voting Result)

Link: SUN's presentation on JPA 2.0 in JavaOne Conference 2007

Watch out for JBI 2.0

Currently, JBI 2.0 specification is in progress. It is expected to be finalized in Q1 of 2008. The JCP reference for JBI 2.0 is JSR 312. [Link to JSR 312]

The scope of JBI 2.0 includes followings:

• Enhancements to facilitate the use of JBI in clustered or distributed environments, principally with respect to administration rather than the clustering/distribution mechanism itself.
• Enhancements to clarify and enhance the use of JBI in a SOA-based approach to the creation, deployment and runtime support of Composite Applications.
• Enhancements to support requirements stemming from WS-Policy.
• Enhancement to support Web 2.0 technologies and usage models.
• Introduction of a Message Exchange handler/interceptor model.
• Enhancements to facilitate performance optimizations by component and container implementers.
• Improved alignment with Java EE (e.g. use of transactions).
• Recoverability of Message Exchanges.
• Improved readability of the specification to clarify the needs of container, component and application developers.
• Alignment with the Service Component Architecture (SCA) specifications (see www.osoa.org) with the goal of making JBI 2.0 a standard Java runtime for SCA .
• Enhancements to support full compatibility with OSGi, without necessarily requiring OSGi.
• Technical issues stemming from implementation experience using the JBI 1.0 specification (e.g. life-cycle of components, error handling, interop profiles, examination of the utility of WSDL definitions for non-Web Services deployed components, component attributes, threading, NIO use, classpath or endpoint activation)

The above list is taken from the proposal of JSR 312. I am more interested in the highlighted items. One of the most important drawback of JBI 1.0 is that its Normalized Message Router (NMR) is not based on a messaging system, but it is an in-memory router. It makes it a single point of failure in the system. I hope that the scope named as "Recoverability of Message Exchanges" address this issue.

Is JBI 1.0 true ESB?

Sun has taken a big leap with new integration specification JBI 1.0. The JCP reference for this specification is JSR 208. The specification is available here for download: JBI 1.0 Specification.

One of the key component of JBI 1.0 is Normalized Message Router (NMR). It routes normalized messages from the source component to its destination using some kind of routing policy.

In a true ESB based architecture or product, the integration is based on a messaging system, which acts as a message broker. The role of this message broker is to provide abstraction between application components. It routes the message from one component to another component, and the components are not aware of location of each other. It provides following functionality:

1. Reliability: In case of failure or unavailability of any component, the messages are persisted in a queue.
2. Scalability: The end point of ESB can be deployed on multiple servers. These servers are linked together with the messaging system. The ESB knows the location and type of services deployed on each server. It uses this information to route the messages to its destination.
3. Security: Transport level or message level security can be provided by the messaging system while transporting the messages from one server to another server.

In JBI 1.0 specification, the Normalized Message Router (NMR) is expected to play the role of message-broker for the ESB. However, the NMR in JBI 1.0 is designed as in-memory message router. It is not based on any messaging system. Therefore, it does not provide all the functionality of the message broker of a true ESB. Moreover, it becomes a single point of failure. In case JBI 1.0 based server crashes, the in-transit messages are lost.  In my opinion, JBI 1.0 based product does not qualify for a true ESB solution.

David Chappell paper : Introducing SCA

David Chappell has written a nice paper on SCA technology: Introducing SCA

The paper explains the concept of SCA and its attributes (components, composite, domains etc.). It includes samples of new SCA components. To learn SCA, this paper is a good starting point, which covers all the basics of this new technology.

The SCA specification defines:

1. A new programming model to create services. For example: creating EJB is a programming model, which the developer defines classes & configuration files following EJB standard. The EJB is deployed in an EJB container, and the ejb-attributes (security, transaction etc.) are defined in the deployment-configuration file. In similar way, SCA defines a new programming model to create classes and configuration files. These classes and configuration files are deployed on SCA runtime (Example: Tuscuny).
   
2. Composition of components: SCA specification a new method to create composite components. This is very much similar to orchestration. The orchestration is defined using BPEL like language. In this case, it is defined in the configuration file of SCA. One component gets reference to another component using inversion of control pattern. So, if one component composes of another, it is defined with annotation  "@Reference". The reference is defined in the configuration file. It is the responsibility of SCA runtime to create the object and make it available to the parent class using getter/setter.

However, it briefly touches some advanced (but important) aspects of any services. Well, this paper being the introduction, the advanced topics is out of scope for this papers. The topics, which I am referring, are:

1. Transaction Handling
2. Security Implementation
3. Development Tools
4. Runtime
5. Monitoring (SCA does not specify standard MBeans)

A new SCA transaction-policy specification (draft version) is published on 3rd December 2007 (Link: SCA Specifications). I have not reviewed it so far.

Is Agile Software Development process effective for large projects?

Recently, I worked on a project following Agile development process. The size of the project was big including a J2EE Web application, integration using an ESB product and Web-Services. The development teams were located at two places in USA and one in offshore (India). Before start of the project, the client decided to use Agile process for this project. I had worked on other development process such as RUP, SDP, Waterfall etc. This was my first experience with Agile process.

Principles of Agile Process

Following principles of Agile Process were followed:

1. The requirements were captured in the form on user stories. There was no formal requirement or use-case document for the entire project.
2. The project was executed in iterations. In one iteration, few user stories were selected for implementation. During selection of user stories for one iteration, it was ensured that the implementation time would not be more than 4 to 6 weeks.
3. The development team, business analyst and testers sit in one big conference room at all three locations. There was one hour of meeting daily to keep track of development, issues and other items.
4. The development started on the very first day. The developers were given the user stories, and list of frameworks/tools to start the project. The developers had marathon meeting to discuss the implementation strategy. They divided the work and started coding.
5. Business analysts and testers worked with developers to assist in understanding the requirements and testing the deliverables.
6. Each iteration delivered a new version of application for deployment in production.
7. The requirements were not static. It could be changed anytime during development.

This process was followed for few iterations, and we started seeing problems with this approach.

Problems with Agile Process and Lessons Learnt

1. No Requirement Document: Agile process recommends to minimize the documents. And, the requirements can be changed by the customer anytime. This creates a lot of confusion to the customer. A lot of time is wasted in discussion with the business analyst in finalizing the requirements. As analyst is not bound to freeze the requirement, the developers are always in the state of uncertainty that the change may be coming soon.
   
   This creates more problems to the project manager specially in case of fixed-bid contracts. Since the scope of the iteration is not defined, it becomes difficult for the project manager to mange the scope of deliverables. Agile process says that "Forget about Change Requests", which is good for the client but not for the consulting/contracting company.
   
2. No Design Document: The developers are expected to discuss the implementation approach and start development as soon as the user story is received. Now a days, the applications (Web or Integration) are so complicated that you need a good framework and abstraction around common functionality. This improves consistency and reuse.
   
   In Agile process, there is no time allocated to develop the common framework to be used by all developers. In absence of the common framework, each developer tends to write code in its own style. This creates huge inconsistency. The problem compounds when the teams are sitting in different locations and different time zones.
   
3. Senior Developers only: The developers are expected to discuss the architecture & implementation strategy, and start programming. This requires that the developers must have good experience in developing similar applications. For junior developers, it was always a challenge to understand all the decisions taken during the discussion and start implementing it.
   
4. No Team Lead / Supervisor: All developers are expected to work on certain deliverables. Usually, there is only one leader in Agile process, who manages the entire development cycle. But, the leaders responsibility is very much similar to a project manager which includes managing the user stories, scope of iterations etc. The developers work on own. In large projects, (as per my experience) technical and QA leads are very important to manage the deliverables. Without a lead, each developers tend to follow its own design patters, coding practice resulting in poor quality, duplicate work and inconsistency.
   
5. Incremental Deliverables: The project is divided into user stories and the life of an iteration is around 4 to 6 weeks. Each iteration creates a version of the application to be deployed in production. If the application is actually put in production and is used by the end-users, it requires migration effort for each iteration. For large projects, the requirements are built incrementally, the database is changed in each iteration. This requires migrating data in the end of each iteration, which becomes a signification effort (but not useful) by the end of the project.
   
6. Managing Large Developers: If a large development team is working, it becomes difficult to allocate work to each developer for a 4-6 weeks development cycle. Few developers get a lot of pressure and few get a lot of time for "Googling/Coffee/Smoking".
   
7. No Task Level Schedule: Usually, the only task defined for an iteration is to implement the user-story. This becomes difficult for the project manager or lead to keep track of the progress. Changing requirement adds more problems to it.
   

Conclusion

Looking at pros and cons of Agile Development Process, I feel that it may be good for small projects or the maintenance/upgrade projects. In these cases, as the framework and other documents/tools are already created, there is a baseline. The upgrade would require minor and local changes, which can be managed very easily and effectively in terms of user stories and small iterations. For large projects with large development team & multiple development sites, Agile Development Process may not be a good choice.