Running background workers on anynines

Background workers are good for you

As time and coding goes, your application will grow and become more and more complex. In such a system you will sometimes have to run time-consuming tasks or tasks that are very computationally intensive. You know what this means – app slows down to a crawl.
Fortunately, this is where background jobs/workers come into play.

With background workers you can save jobs in a queue and process them later without blocking your current thread. You can even run several jobs in parallel!

Another advantage of background jobs is the better user experience. When you execute time-consuming jobs in the background, requests return immediately and users don’t have to wait and get angry at you. They just browse your website and get notified when the job is done.

Since Rails 4.2 it’s very easy to use different background workers in your application. In this version they introduced Active Job. That’s a framework for declaring jobs and making them run on different queueing backends. It provides a place for jobs in your rails apps and ensures that you don’t have to worry about differences between various background workers like Resque, Sidekig or DelayedJob.
So, how do we use background workers in our apps?

Workers at work

In our example app we’ll use Rails 4.2 to benefit from the features of Active Job. As the background worker we’ll use DelayedJob, a database-based asynchronous priority queue system.
The finished app can be found here. It’s a simple blog app from the rails guides, but in our example we use a background job to delete comments.
When you have set up the base app we can start implementing the background job.

Integrate DelayedJob

First, add the gem to your Gemfile:

gem 'delayed_job_active_record'

And install it with:

bundle install

As DelayedJob is based on a database, you have to create the required table:

rails generate delayed_job:active_record rake db:migrate

Now let’s make sure that our app uses the desired background worker – in order to do that, specify the adapter in config/application.rb:

module Rails4WorkerExample
class Application < Rails::Application
config.active_job.queue_adapter = :delayed_job
# comments and other things
end
end

After you’ve configured everything correctly, let’s create your first job.

Create your job

Rails has a built-in generator for creating jobs. To create a job run in your console:

rails generate job delete_comment_job

The generated jobs are located in app/jobs. A job looks like as follows:

class DeleteCommentJob < ActiveJob::Base
queue_as :default 

def perform(*args)
# Do something later
end
end

Now update the perform method so that it deletes a comment:

def perform(comment_id)
# Do something later
sleep 5
comment = Comment.find comment_id
comment.destroy end

Your job is ready! We just have to use it in our app.

Use your job

As the job deletes comments, we’ll use it in the destroy method of our comments_controller:

def destroy
@article = Article.find(params[:article_id])
@comment = @article.comments.find(params[:id])
DeleteCommentJob.perform_later(@comment.id)
redirect_to article_path(@article)
end

To add a job to the job queue, call perform_later on our DeleteCommentsJob class. At this point, DelayedJob will run the job for you in the background. Just sit and enjoy your coffee :)
Aaaand we’re almost there – the last thing you have to do is deploy the app on anynines and try it out.

Let’s deploy! Enter anynines

Let us deploy your app on anynines. First, you have to select the anynines api:

cf api https://api.de.a9s.eu

Then login with your credentials and select your org and space:

		cf login
		cf target -o <your org_name> -s <your space_name>

Before you can push your app to anynines, you need to create the required service. In this case it’s mysql:

		cf create-service mysql Pluto-free <SERVICE NAME>

After that we create 2 manifest files: app_manifest.yml for the app and worker_manifest.yml for the worker.

app_manifest.yml
---
applications:
- name: <APP NAME>
  memory: 512M
  instances: 1
  host: <APP NAME>
  domain: de.a9sapp.eu
  path: .
  buildpack: https://github.com/cloudfoundry/heroku-buildpack-ruby.git
  services:
    - <SERVICE NAME>
worker_manifest.yml
---
applications:
- name: <WORKER NAME>
  memory: 512M
  instances: 1
  path: .
  no-route: true
  command: bundle exec rake jobs:work
  buildpack: https://github.com/cloudfoundry/heroku-buildpack-ruby.git
  services:
    - <SERVICE NAME>

They may look almost the same but have some important differences. In the worker_manifest.yml we set the option no-route to true so that no route is assigned to our worker. With the command option we can specify our custom start command for the worker.
Both need the same mysql service, though – otherwise our worker and app won’t work together.

Now’s the time to deploy the app and worker to anynines:

cf push -f app_manifest.yml
cf push -f worker_manifest.yml

The app is now available at <APP_NAME>.de.a9sapp.eu. To see the effect of our background job, do the following:

  • create an article and write some comments,
  • delete a comment,
  • and refresh yout page after 5 sec – voila! the comment is gone

Our brave background worker is doing its job perfectly. And you know how to configure your rails app to use background jobs and handle complex work in the background!

Workers work wonders

When you have a time-consuming or complex task you should definitely use background workers. Your users will thank you for the better user experience – they won’t wait while their tasks are executed and the requests will return immediately. Plus, background workers also make your app more scalable and easier to manage. With Active Job you can now easily declare jobs and run them on different queueing backends.

It all boils down to one thing – just add more workers if you need them!

Hope you liked my blog post – you can tweet directly. Follow @anynines for more hosting news and tutorials!

anynines is going live – best time to give it a go!

anynines goes out of betaThis is an article about anynines – from the very beginning to the final going out of beta.
If you want to learn about anynines history, keep on reading – if you’re eager to know what will happen in the future, hop to the last paragraph of this post!

Why we founded anynines

anynines was founded in April 2013 at the Avarteq Headquarters in Saarbrücken, Germany.
From the beginning hosting was an essential part of Avarteq and we gained first-hand experience in this field with Enterprise-Rails, a hosting brand featured by Avarteq.

With that knowledge, it became clear to us that we want to take hosting to a next, revolutionary level – establishing an European Platform as a Service (PaaS) based on the Open Source software Cloud Foundry.

We gathered in a first brainstorming session to find a name for our new product. We always push hard for 99,999…% availability and application security in hosting, so we decided on the name anynines shortly before our brains set on fire. We never thought we’d spend so much time on a brand name alone!

anynines was born and we set out into the future of Cloud-Hosting. We started, naturally, with a beta.

What we did during the beta phase

Firstly, we set up our team. In the first days our team had a size of three to five people but over the months more and more new members joined the team. The team members these days were Julian Fischer, the founder of anynines and one of the CEOs of Avarteq, Oliver & Julian for the development part, Nico for admin tasks and me for the marketing.

For final copywriting and design we worked together with Mirek Woźniak and Mariusz Cieśla who both did a fantastic job and we could welcome Floor Drees who was responsible for the growing community. 

Why did we choose Cloud Foundry?

We compared different emerging PaaS projects to have an overview over the market and believed in the concepts of Cloud Foundry. We believe in the power of Open Source standards which also enforced our decision towards Cloud Foundry as a basis for anynines.

Our estimation proved to be correct and Cloud Foundry has developed to the market leader in Open Source Platform as a Service frameworks with a rapidly growing community and the well funded Cloud Foundry Foundation behind the project. To manage our infrastructure automation we used BOSH from the beginning. In our vision anynines should support both – private and public usage scenarios.

Challenges

Hence Cloud Foundry was our first choice for establishing anynines. There were some obstacles to overcome.

We had to stabilize the runtime and weigh out Cloud Foundry components equally. We also had to implement an invoicing integration, a customer panel and fill other feature gaps.

Service development

Because the existing community services unfortunately were outdated and not high available as we needed them to be. For our vision of anynines we have to create highly available updated service solutions on our own. These will be available soon.

Establishing the anynines community

We had the PaaS and the IaaS but as we all know – an awesome idea is nothing without people learning about it and joining the movement. So we had to establish social media channels, a support board, marketing strategies and and and…

A big issue for us was to get in touch with the people we want to provide anynines to as an alternative European service to PaaS providers from the US. So we started sponsoring meetups, workshops and conferences and visiting them.
So our anynines community began to grow and today a solid group of anynines enthusiasts is helping us by giving important feedback, writing tutorials and HowTos and even mentioning us in blog articles.

Out of Beta our biggest challenges

So here we are now – going out of beta and providing a functional European Platform as a Service solution for application developers. In addition we work hard with our partners to support them in creating their own private PaaS solutions based on the anynines software stack.
But before this day we had some big challenges to cope with. For example we switched from a rented VMWare infrastructure layer to a self hosted OpenStack IaaS layer.

What the “out of beta” means to you

If you’re already an anynines user you’ll automatically enter a 30 day trial period for your application instances with the same quota as during the beta phase. The only thing you have to do – if you didn’t already – is to deposit your payment data. This is important because after these 30 days the resources you use with your account will be charged for. In case you want to view a list of used resources in detail – just have a look at the customer panel.

That brings the future

We are focused on bringing new features regularly to you. To give you a short view about what we’re working on we’ll go deeper into the anynines service development plan. Our goal is to exchange each community service one after another. We’re also working on service instance connections via client tools from outside of the cloud.

Right now we are testing our MongoDB service implementation internally which includes high available Replica Sets, the latest MongoDB version and dedicated service instances.
In addition we are preparing internal PostgreSQL test runs for our new service implementation which provides a high available cluster solution, the latest PostgreSQL version and dedicated service instances.

We also work on developing a Redis service. Not to forget the Diego (next generation application executor) integration we want to provide to you as soon as possible.

As you can see there is a lot going on so you can be curious what we’ll bring to you in the future. If you have not worked with anynines yet – just try it out.

How to deploy on anynines using Typesafe Activator

TL;DRTypesafe-Activator

A short version for the impatient.

I assume you’ve got git, Typesafe Activator and an anynines user account.
Clone the source:

$ git clone git@github.com:anynines/a9s-java-pingpong.git

Skip to “Push it to the cloud” below.

Typesafe Activator is cool and here’s why

I know Java since the mid-nineties and to be honest it’s my favorite weapon of choice when it comes to coding.

I’ve done much web development in the past and when I heard about Play! Framework many years ago I was pretty excited. Take the philosophy of Ruby on Rails adopt it to Java and you’ve got Play. Play evolved over the years and is now wrapped inside Typesafe Activator. In fact, I stumbled upon Activator because of Play. Activator can do way more than I use it for but it lets me get things done quickly.

Lately I got to know about anynines and one of the first things I wanted to do was to deploy a Java application to anynines. Nothing fancy, it wasn’t even a web app – just a kind of “Hello World” app, but it serves well to show how to combine Activator and anynines.

Prerequisites

To follow my steps you’ll need a working JDK on your machine as well as Typesafe Activator, the cf CLI and a anynines user account.

Creating the basis

Let’s start with setting up a fresh Activator project first:

$ activator new

Fetching the latest list of templates...

Browse the list of templates: http://typesafe.com/activator/templates
Choose from these featured templates or enter a template name:
1) minimal-akka-java-seed
2) minimal-akka-scala-seed
3) minimal-java
4) minimal-scala
5) play-java
6) play-scala
(hit tab to see a list of all templates)
> 1
Enter a name for your application (just press enter for 'minimal-akka-java-seed')
> a9s-java-pingpong
OK, application "a9s-java-pingpong" is being created using the "minimal-akka-java-seed" template.

Now cd in the freshly created directory and run:

$ activator

You’re now inside the Activator shell where you may issue commands.

Modifying the application setup

I usually use Eclipse to do Java coding, however there is an Eclipse Plugin for SBT that we can use to have SBT (yes, we are in Scala-land by now :) generate Eclipse project files from our Activator project.

Additionally, we’ll need a plugin for SBT that allows us to package our final app in a way that is suitable for the cf CLI in order to deploy our app to anynines. To accomplish that task we’ll use the SBT Native Packager plugin.

Modify “project/plugins.sbt” (create the file if it doesn’t exist) as follows:

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.0")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "3.0.0")

Append the following lines to “build.sbt”:

lazy val root = (project in file(".")).enablePlugins(JavaAppPackaging)

EclipseKeys.withBundledScalaContainers := false

Restart Activator (STRG+C) for the changes to take effect.

Modifying the code

Fire up your favorite code editor or let SBT generate Eclipse project files while you make yourself coffee. From within Activator run:

> eclipse

Now you may use Eclipse’s “import existing project” function.

We want our Actors to constantly play Ping-Pong, so we’ll remove the counter in PingActor and modify the onReceive method as follows:

public void onReceive(Object message) throws Exception {
  if (message instanceof Initialize) {
    log.info("In PingActor - starting ping-pong");
    pongActor.tell(new PingMessage("ping"), getSelf());
  } else if (message instanceof PongActor.PongMessage) {
    PongActor.PongMessage pong =
      (PongActor.PongMessage) message;
    log.info("In PingActor - received message: {}",
      pong.getText());

    // schedule ping to be send in 1 second
    getContext()
      .system()
      .scheduler()
      .scheduleOnce(Duration.create(1, SECONDS),
        getSender(),
        new PingMessage("ping"),
        getContext().dispatcher(),
        getSelf());
  } else {
    unhandled(message);
  }
}

Now modify the onReceive method of PongActor as follows:

public void onReceive(Object message) throws Exception {
  if (message instanceof PingActor.PingMessage) {
    PingActor.PingMessage ping = (PingActor.PingMessage) message;
    log.info("In PongActor - received message: {}",
      ping.getText());

    // schedule pong to be send in 1 second
    getContext()
      .system()
      .scheduler()
      .scheduleOnce(Duration.create(1, SECONDS),
        getSender(),
        new PongMessage("pong"),
        getContext().dispatcher(),
        getSelf());
  } else {
    unhandled(message);
  }
}

Feel free to test the app locally by executing

> run

inside Activator.

It’s alive! Push it to the cloud

Since we have a working application now, let’s run it in the cloud. We’ll use the SBT Native Packager plugin to have everything we need packed for a successful deploy. Open Activator and run:

> universal:packageBin

Relative to the root directory of the project we get a ZIPped archive:

target/universal/a9s-java-pingpong-1.0.zip

Make sure that the target for the cf CLI is correct:

$ cf api https://api.de.a9s.eu

Then login using your credentials selecting ORGs and spaces as you like:

$ cf login

Now let’s push our application to anynines:

$ cf push pingpong --no-route \
  -b http://github.com/cloudfoundry/java-buildpack.git \
  -p target/universal/a9s-java-pingpong-1.0.zip

Since we do not provide a HTTP interface (remember: our application does not interact with the outside world), we tell cf with the no-route option to avoid route creation (and with that the corresponding health check).

Voila! Let’s check if our application is running:

$ cf app pingpong
Showing health and status for app pingpong in org ***_*** / space test as ***@***...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls:
package uploaded: Mon May 4 13:57:33 +0000 2015

       state   since                    cpu    memory           disk        details
#0   running   2015-05-04 03:58:13 PM   1.4%   148.5M of 256M   125.3M of 1G

Looks good! Now watch the Actors play Ping-Pong:

$ cf logs pingpong

Well done!

You’ve just successfully deployed a plain Java application (using main(.) as PEP) to anynines. Plus, we’ve used Akka to have two Actors play Ping-Pong.

With that knowledge we are ready to start building asynchronous messaging applications that are heavily scalable using Java, Akka and anynines. But that’s for the next episode – as for now, stay tuned to our twitter and shoot me an email if you’d like to talk about Scala-land :)