REST Web Service API Guidelines

When building web services, one of the primary benefits of using REST over SOAP is the intuitive nature of service interfaces.  However, this simplicity of interface can very easily be eroded.  Below are some suggested (and hence flame proof) guidelines that could be followed to ensure continued interface simplicity:

Resource-Oriented

• REST is resource-oriented, not service-oriented.  Resources are nouns, not verbs.

Addressable

• Every resource must be addressable by means of at least one URI (name).  Names must be meaningful.
• Clients cannot access resources directly - they deal in representations of that resource (e.g, XML, JSON, ...)  
• Resource representations would ideally be addressable (so that they can be passed around as URIs, eg. /rest/bookmarks.xml)  The use of HTTP accept headers to specify representation is however acceptable, but should be provided as well as the addressable URI.

Read More

Microservices: New spin, old hat?

This year's QCon 2015 conference in London was abuzz with talk of microservices, and if you, like me, have seen protocols and standards like CORBA, RMI, J(2)EE and SOAP come (and go), you would be forgiven for sighing and thinking "what's new"?  However, it seems that Google, Amazon, eBay and others would beg to differ.

In a nutshell, microservices:

• Are small (they fit in your head)
• Run in their own process
• Are independently replaceable, upgradeable and deployable via automated/test deployment mechanisms
• Are backwardly (and forwardly) compatible
• Communicate with each other via lightweight mechanisms, typically an HTTP resource API
• Are organised around business capabilities.  "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." -- Melvyn Conway, 1967
• Involve products, not projects
• Require decentralised governance
• Require decentralised data management (independent persistence)
• Automate their infrastructure (e.g. AWS)
• Are designed for failure of dependent services
• Prefer asynchronous calls to synchronous ones
• Involve evolutionary design
• Are fully developed, owned and supported by single teams
• Compete with each other in a market-place

Much of this will be familiar, but what is new and very interesting is the degree to which microservices apply the familiar principles of loosely-coupled, encapsulated software components to the organisation developing and managing those components.

Traditionally structured companies will balk at all of this, and perhaps dismiss this as yet another architectural fad, but I'm not so sure.  Check out these two excellent videos on Spotify's Engineering Culture.







Further reading:

• Martin Fowler at al
• Microservices.io

Read More

Supporting Your Public Web API

No matter how beautiful and well-designed your new web API is, unless it is well supported, you may find that you become a victim of your own success.  Below are 4 suggested key elements of a supportable web API, based on my experiences at IG.


1. Documentation, documentation, documentation!

Unless it is easier to refer to your API documentation than to contact you, you will be contacted.  Don't assume anything is obvious.  Document your API in detail. Define your business terms.  Explain common interaction patterns.  Provide client code snippets and working samples in a variety of language bindings.   Provide a FAQ.

Consider providing a client SDK (Yes,I know REST is meant to be easy, but the reality is that unless your client is already a developer, they will struggle.  Make it easy for them to use your API.)

Consider providing a moderated forum to allow clients to answer each other's questions.  Provide an interactive API explorer allowing clients to try out APIs and see the raw HTTP requests and responses.


2. First and Second Line Support

Provide a "contact us" form instead of direct phone numbers.  The page should encourage clients to refer to API documentation, FAQ, and forum (if one exists).

Provide first line, "customer sensitive" support to answer common queries, or at least to ensure queries are well formulated before passing onto second line support.  If your organisation already has a customer services help desk, use them! They may balk initially at the perceived technical nature of the support, but should be helped past this by through training and provided access to simple debugging tools, e.g. the API explorer or a facility to search service log files.

Provide dedicated second line support via a rota consisting of the developers who built the API.

Maintain a searchable log of frequently answered questions (and answers).  Update the public FAQ from this log.


3. Effective error handling

Give API clients as much error information as possible in request responses.  If inputs are incorrect - tell them why.  If error codes are used - ensure these are documented.

Log every request in detail, including request authentication headers and body (subject to client privacy constraints).  Use a transaction id pattern to allow the flow of API requests to be tracked across multiple back-end systems.  Provide easy access to system logs, e.g. via Splunk, which provides a searchable index over a number of configured log files.  Splunk additionally allows requests to be charted and management dashboards to be created.  Senior management will want to know just how popular the API is.


4. Provide a barrier to entry

If, despite all of the above, the demand for your API is still overwhelming, consider:

• throttling your API, with chargeable tiers for increased quotas
• requiring that a verified account to be opened before an API can be used
• providing support to valued customers only (or on a subscription basis).

Read More

What is SOA?

What is SOA?  SOA or Service-Oriented Architecture is a software architecture pattern in which applications or systems are constructed from underlying (and usually distributed) software services that conform to a specific set of characteristics, namely:

1.Contract based
2. Location transparency
3. Autonomous
4. Abstract
5. Reusable
6. Composable
7. Stateless
8. Discoverable
9. Extensible
10. Loosely coupled

The primary goal of SOA is software development agility, i.e. the ability to respond the change easily, and cheaply, thus allowing businesses to rapidly respond to changing markets.

Services are typically (but by no means exclusively) implemented as web services, i.e. they operate over the ubiquitous web HTTP protocol, and are implemented either using XML-based SOAP or the lightweight (and more popular) REST paradigm.

The highly distributed nature of this architecture has resulted in a need for a runtime platform (the Enterprise Service Bus, ESB) to help manage the operation of these services, but also to handle complex enterprise integration scenarios involving multiple (and often legacy) platforms, protocols and security models - products like Oracle Service Bus or the open source Mule ESB.

In addition to being an integration solution, the service-oriented pattern also offers an opportunity for easier implementation of high-level business processes using Business Process Management (BPM) solutions that are often offered as part of the ESB product suite.

Service-oriented architecture is highly relevant to the rapidly growing cloud computing model where applications are built in the cloud as orchestrations over public cloud-based APIs, and we are seeing ESB vendors like WSO2 and MuleSoft in or moving into this space, offering cloud-based versions of their products, as well as API management products.

Some will argue that SOA is not new, and that distributed programming involving services has existed for a long time, and they are right.  However those early efforts stumbled on the hurdles of platform incompatibility and standards complexity.  Now, with ubiquitous, and easy to use protocols like REST, web service API development is growing rapidly, and the integration possibilities seem endless.

(What is SOA, Service Oriented Architecture)

Read More

REST Web Service Paradigm Shift

Moving your Service-Oriented Architecture (SOA) from RPC style (web) services to SOAP document-oriented services, or REST resource-oriented services requires a difficult mental paradigm shift.  The same kind of shift that was required for monolothic and procedural programmers to move to object-oriented programming styles.  This change is not easy, and not all will pass muster.  Failing to make the change will mean the difference between Just a Bunch of Web Services (JABOWS) or a fully-fledged SOA.

Ensuring that this mind shift does happen will require good organisation and strong, well-articulated guidance and governance.  Until the new pattern becomes habit, people will very quickly slip into old habits. 

Below are some guidelines and implementation suggestions from this blog:

• Key REST service guidelines
• REST service naming guidelines
• REST service error handling
• Service design principles
• A plea for HATEOAS
• HTTP error code guidelines

Read More

REST Web Service Spring Error Handling

The Spring framework provides excellent support for building REST web services in a service-oriented architecture (SOA), as I have demonstrated in a previous post, but what is not immediately obvious is how best to deal with errors. Instead of throwing raw exceptions to the client, one would prefer to return a well-formed response consisting of an appropriate HTTP status code and a meaningul response body containing a structured error message in the response form (JSON, XML, ...) requested by the client.

For example, assuming you have a basic REST endpoint such as:

 @RequestMapping(value = "/samples", method = {RequestMethod.GET})
 @ResponseBody
 public SampleList findAllSamples() throws SampleException {
  return new SampleList(service.findAllSamples());
 }

It is good practice in general to not let implementation exceptions leave the service tier, so you might use aspects to catch these and rethrow them as service exceptions.  The code below catches a ReferenceNotFoundException from the data access tier and throws a service exception with the reference number and setting the desired http status code.  Another good practice is to avoid sending free text error messages to a service client, and instead to send structured messages that can be dealt with or rendered as the client sees fit.

@Aspect
@Component("exceptionHandler")
public class ExceptionHandler {

   /**
    * Exception handler for ReferenceNotFoundExceptions
    * 
    * @param nfe the exception being handled
    */
   @AfterThrowing(pointcut = "execution (* com.sample.service.*.*(..))", 

                                              throwing = "nfe")
   public void handleNotFoundException(ReferenceNotFoundException nfe) {
      throw new SampleException(
         new ReferenceNotFoundError(nfe.getReference()), 

                               HttpStatus.NOT_FOUND);
   }

Now to map the service exception (SampleException) to a meaningful HTTP response using the Spring 3 @ExceptionHandler annotation:

   @ExceptionHandler(SampleException.class)
   @ResponseBody
   public SampleErrorList exceptionHandler(SampleException se,
                 HttpServletRequest req,
                 HttpServletResponse res) throws IOException {

      res.setStatus(se.getHttpStatus());

      return new SampleErrorList(se.getErrors());

   }
 

It is also possible to set the HTTP status code using an annotation, but in this example the exception determines this, so the handler sets it programmatically.

See here for a downloadable working sample.

REST Web Service Spring Error Handling

Read More

Funny SOA Joke?

“Knock, knock.”

“Who’s there?”

... very long, expensive pause ....

“SOA”

Read More

Establishing ROI on your SOA, EDA, ESB, BLAH, BLAH

Establishing ROI on your SOA, EDA, ESB, BLAH

Read More

Applying ITIL v3 to SOA

I wrote at length in 2008 on applying ITIL (v2) to SOA, and it is interesting to see how much of that still applies to ITIL v3, and in fact even more so, given the focus on service.

To recap:

"Service-oriented architecture (SOA) proposes a model of software as a distributed network of cooperating services, in contrast to the traditional, more monolithic application model. Operationally managing such applications requires a sophisticated management organisation and operating framework that are capable of defining and sustaining service levels to customers across the enterprise.

ITIL is the widely adopted framework for service management, defined as the management of an IT infrastructure of hardware, software, communications equipment and facilities, documentation, and skills used to provide the required service at the required level of quality."

Though the ITIL definition of service is much broader than that of SOA, the ITIL provides a useful framework for thinking about how to build and manage your SOA.

Service Strategy

A well defined and clearly articulated service business strategy is crucial to the ongoing success of a Service-oriented Architecture.  It needs to be "defined" so that the architecture evolves in the desired direction.  It needs to be "clearly articulated" so that during the long journey people remember why the process was started.  Your strategy will answer questions like:

• Why are you building software as services?
• What is the business case?
• What is the goal (in bothe business and technical terms)?

Service Design

The ITIL concept of service design expands the traditional software service analysis design activity to include the consideration of the following:

Supplier Management :  "No man is an island", and this applies especially to SOA, where the number of "suppliers" that services rely upon to fulfil their function increases dramatically over traditional, monolothic software architectures.  Supplier management is about understanding which services your services rely upon, who owns those services,  and defining contracts and SLAs with those service providers.

Service Level Management : Having defined contracts with service consumers and suppliers, service level management is about monitoring and enforcing that SLA, dealing with exceptions, and communicating effectively with both.

Service Catalog Management : Your service catalog or repository or registry or whatever you want to call it is that place where questions like the following are answered: What are your services?  Where are they? What different versions are there? Who uses them?  What SLAs are defined?  Where can people find out about services?

Availability Management : Availability management addresses questions around ensuring that your services can meet their SLAs, considering the availability of services during planned or unplanned service outages, unexpected increases in volumes, etc

Service Transition

The initial build of your services will only form a tiny part of the overally lifetime of the service.  Managing the potentially frequent changes to your services forms the focus of the ITIL service transition discipline.

Change Management : Change management is about being able to introduce change with minimal disruption to service consumers.  A versioning policy will form a key part of this process, that is the ability to introduce new service versions without disrupting existing clients, as well as defining processes to enforce managed migration to the newer versions.

Knowledge Management: Effective decision making and service support requires information such as who is using your services, how are they being used, etc

Release and Deployment Management :  SOA requires continuous deployment methodologies in order to satisfy the SLAs that govern the linkages between services.  Consideration needs to be given to how multiple service versions may be run concurrently, and the how deprecated services will be managed and phased out.

Service Testing and Validation: SOA requires automated testing to allow rapid regression testing

Configuration Management System: SOA requires that services are versioned, and readily visible to the client community.

Service Operation

Operating a complex network of interoperating services is more complicated than monolithic architectures and needs

Event Management: monitoring services, alerting and triggering responses

Incident/Problem Management : service incident logging and classification, knowing what steps to take, who should do what, SLAs for completion, escalation procedures, client notification procedures, evidence capture, reviews,

Request Management : providing channels for clients to request an receive services, information to customers about service availability and procedures for obtaining them, sourcing and delivering services, assist with general information, complains and comments

Access Management : managing access to service to authorised users and nobody else, monitoring

Read More

SOA or Just a Bunch Of Web Services (JABOWS)?

A Service-oriented Architecture (SOA) is more than just a bunch of web services (JABOWS), but which do you have?

• Do you have a place where services can be discovered, learned about and tried out?
• Are your services being reused, sometimes in unexpected ways?
• Are your service clients viewed as customers?
• Are you able to upgrade service implementations and interfaces without any disruption to your service clients?
• Do you know how your services are being used?  Do you have service-level agreements that are being monitored and enforced?
• Do you have a service charging model?
• Are your developers enjoying and being rewarded for reuse?
• Do you have processes for regulating the usage and change of services?
• Are you able to provide dynamic capacity management?
• Do you have service development, interface and contract standards?
• Are your services secure?
• Do you have automated service regression testing?
• Does your business understand the need to protect its service assets, and to curb its appetite for tactical change?
• Could your services be easily exposed to the Cloud?

I think the last question is probably the most incisive.  If the answer is no, then you are just doing a bunch of web services, no matter what lip service you pay to the other questions.

Read More

SOAP vs REST web services

Comparing SOAP to REST for building web services is a bit like comparing apples with pears: SOAP is a protocol, REST is a pattern, and you could, if you felt so inclined, write RESTful services using SOAP.  However, when people ask this question, they usually mean: SOAP (as in WSDL, perhaps RPC style, WS-* standards), as opposed to REST (as in JSON/XML over HTTP)

Assuming you mean this, below are some pros and cons for each, but note this: REST is increasingly becoming the de-facto standard across the web, being used by the big public API providers in preference to SOAP. I reckon it's just a matter of time before this question will just go away.

Read More

SOA - Governance Principles

Definition:-

Governance is the application of controls to processes that change corporate assets, with a view to ensuring that those assets comply with (or move towards compliance with) with a set of corporate goals

Governance Needs Balance:-

Given the necessary overheads of governance, a balance needs to be struck between too much governance (which will cripple processes), and too little governance (which will result in chaos).

Governance Needs A Raison D’être:-

Governance only makes sense against a backdrop of well-defined drivers and goals. For example, an organisation may decide to standardise on a particular architecture, or tool set, for reasons of reduced total cost of ownership. Governance would be used to drive or direct changes to current assets in compliance with these goals.

Governance Must Be Effective:-

Effective governance will impose light-weight, thorough and widely-accepted controls around the changes being applied to those assets. Governance that is seen as pointless, or a hindrance to progress, will be resisted and ineffective.

Read More

Enterprise Architecture or Just A Bunch Of Developers?

In the early days Service-Oriented Architecture (SOA) we were told that SOA was not Just a Bunch of Web Services (JBOWS).  In a similar vein I would like to suggest that in order to build an SOA you need more than Just a Bunch of Developers (JBODS); you need an Enterprise Architecture.  I'm assuming of course that you have a medium to large enterprise to work with.  Small companies will be able to get a way with a single, highly cohesive and agile SOA team, but this does not scale well.

So what do I mean by Enterprise Architecture?

"The Only Constant In Life Is Change." 

- Heraclitus

For a corporation to be successful it must adapt to changing business markets and technological landscapes.  Managing change is crucial to the corporation’s ability to maintain its long-term market agility.

Read More

Web Service API Management

Service-oriented Architecture (SOA) web service API operations management requires a sophisticated service management organisation which is capable of defining and sustaining service levels to its customers across the enterprise. SOA without the service management will result in greater systems fragility and chaos.

ITIL is the widely adopted framework for implementing service management practices and so it figures that it would have relevance to SOA. Under ITIL services must become managed and supportable enterprise assets. This has the following requirements for an organisation:

Read More

Public Web Service API Management Solution

So, as part of your Service-oriented Architecture (SOA) strategy, you've decided to join the Web 2.0 rush to publicise your web service APIs and are wondering about how you might manage this.  Do you build something or are there 3rd party products to help?  The buy-vs-build decision is always a tough one, but you will need to consider your core business and competencies, as well as the product's support for the following:

Partner website - a public portal where consumers of your services will be able to register, obtain service keys, documentation, support, raise issues, join mailing lists, subscribe to RSS updates, download client libraries etc.

Key management - provision and management of API keys and ensure those are supplied with calls.  Support for OAUTH

Read More

Extensible Web Service API

The only constant in life is change, and this is no different for Service-oriented Architecture (SOA). Services must be built with this in mind: that they will have to adapt to new or changing requirements. Extensibility is the ability for services to adapt whilst preserving existing consumer contracts.

We would expect the following types of changes to services to have no impact on other consumers:
- internal source code changes
- interface changes
- take on of new consumers
- environmental changes

Read More

SOA API ESB SAAS PAAS Web 1.0 Web 2.0 Web 3.0 Web 4.0

Interconnectivity is great, isn't it?  :)

Joking aside, we're still trying to recover from the rampant spaghetti architectures we have let evolve in our enterprises.  Let's not repeat these mistakes as we seek everything in the web to be interconnectable.  Cloud ESBs are an interesting development in this space, since they offer the promise of easy integration as well as loose-coupling, which surely has to be key to all of this.

With great power comes great responsibility...

SOA ESB SAAS PAAS Web 1.0 Web 2.0 Web 3.0 Web 4.0

Read More

What is Service-Oriented Architecture (SOA) - Competitive Services

I have been reviewing ITIL 3 with a view to updating my article on Service Management, and came across the notion of competitive services.

This is a very common notion when one looks at services in the usual, non-SOA sense where services are being offered in a market place and have to compete, but does it apply to SOA?

Well, clearly if you are looking at services offered in the internet by companies engaged in Cloud computing and such like, e.g. Google or Amazon, the answer is yes, but what about your typical corporate enterprise landscape?

Read More

What is SOA - Contract based

Software in general is required to conform to a defined functional interface, and services are no different in this respect. For example, web service interfaces defined in WSDL (Web Service Definition Language) allow the definition of data types, input and output messages, operations, invocation protocols, and even the location of services. WS-Policy may be additionally used to define additional elements of the interface such as security policies.

Read More

What is SOA - Service Characteristics - Reusable

A service is said to be reusable if it can be used in more than one context, even in contexts for which it wasn't originally designed.

So how do we achieve this noble goal?

1. Business domain modelling

If you understand your business domain and model services according to that domain your services stand a better chance of being reused than if they were built in isolation to localised requirements. Business process modelling should drive the requirements for services.

Read More