Concept 5

Concept is now 11 years old. I manage to find less and less time to write here.

A lot of new features were added, like video API (H.264 and HEVC), more async APIs, a the main feature for Concept 5.0 is TinDB, a distributed No-SQL database system, written in Concept. Also, Concept 5 will use about 30% less memory.

The main new feature of Concept 5.0 is the fact that will drop the native desktop client. HTML5 is now mature, and there is no need for Concept Client for desktop. It will continue to have Concept Client iOS and Android native versions.

I've been working to my first big-data application, a communications application, written entirely in Concept. You can even video call me from the browser, by clicking here [call Eduard]. Every line of code is written in Concept.

I've redesigned parts of the Concept Framework for better asynchronous support.

As I've said before, the new feature in Concept 5 is TinDB.

The APIs are not final (may be modified, if needed), so this is a pre-preview. The code above inserts a Student object into test_collection, and the database server performs a Wolfram Alpha query upon insert. TinDB has stored Concept procedures.

As usual I've fixed a lot of bugs (a few of them critical). In the next few weeks I will post the entire source code on github.

A redesign for GyroGears apps is in progress, in about two weeks everything should be ready. Gyro apps are incredibly stable, so it will be mostly a face lift, in order to be more HTML5-friendly.

Now I'm working almost exclusively on TinDB and Concept Client 5.0 JS.

Workers, speech recognition and Concept 4.1

Concept 4.1 brings new optimizations, starting with recursive calls (around 50% more efficient), less memory usage for class objects ( - 50%), memory pools for objects, and workers - efficient threads that can be used by the standard Concept Core (no need to use the inefficient multi-threaded core).

Speed and memory optimizations

When testing using the binary tree benchmark (found here: http://benchmarksgame.alioth.debian.org/u64/program.php?test=binarytrees&lang=yarv, and here: http://benchmarksgame.alioth.debian.org/u64/program.php?test=binarytrees&lang=php)

PHP 5.6: Memory used - 369M to 1200M - 345 seconds
Concept 4.1: 79M to 269M - 170 seconds
Ruby 2.2.1: 176M constant - 87 seconds

Ruby is faster on this benchmark only, because it excludes the Concept JIT, how ever, when the JIT is active (ex: mathematical functions and/or loops, based on the Sieve benchmark, concept is about 20 times faster than PHP).

For the fannkuch redux benchmark (foudn here: http://benchmarksgame.alioth.debian.org/u64/program.php?test=fannkuchredux&lang=yarv)

Ruby:

Pfannkuchen(11) = 51
Time: 87435 ms

Concept:
Pfannkuchen(11) = 51
Time: 40401 ms

Ruby seems faster than Concept only on recursive calls, but I'm working on that. On the fannkuchen benchmark, Concept is more than 2x faster.

On Sieve benchmark, Concept scores 8040 iterations/second, while V8 scores on the same machine about 20000, but it uses far more memory. For now, I think that Concept has the optimal balance between speed and memory usage.

Concept Workers

When dealing with multit-hreaded applications, Concept MT core is significantly slower compared to the standard core. This is because it uses a global semaphore for synchronizing threads. The best alternative were the green threads, which are cheap, fast and don't need synchronization. The main drawback is that a blocking call, will block all the green threads. The alternative is the use of a worker.

A worker is a thread that runs in a different context, and exchange data via various APIs with the main thread.

Consider the following example:

Notice that the ChildWorker constructor simply waits for data (via Worker::Pending) and notifies the result via Worker::Result. I was inspired by the javascript web workers, with one exception: the worker runs in the same application (no need for external files).

Speech recognition

Concept 4.1 brings a wrapper for the pocketsphinx library. To see it in action, check the speech recognition example here.

As usual, lots of bugs are now fixed, new APIs for SIP protocol and DTMF, and a more advanced profiler, capable of showing the memory usage,

Concept 4.0

Concept 4.0 stable is finally released. A lot has changed, I think it has the biggest code rewrite since I've started. Is the fastest Concept core, the most memory efficient until now. On Windows it uses dlmalloc as an efficient memory allocator (replacing the standard malloc). It also uses memory pools for variables. It uses shared workers for handling multiple users in the same process, reducing the memory used, and scaling almost linearly. It has an advanced memory compensation engine, for better application feel on high-error connections. It introduces lots of new APIs, from TLS sockets, SSH and wkhtmltopdf/image to loopers (functions called when the main loop is idle).

Concept Application Server 4.0 is entirely written in Concept, using only green threads (non-blocking I/O). When compared with previous C/C++ version (3.0), it uses far less CPU cycles and memory.

Besides the core optimization, the most powerful new feature is the new JS client - Concept Client 4.0 JS (in early beta for now). Now every Concept application ever written may be run directly in a HTML 5 browser. It is tested with Chrome, Firefox, Opera and Internet Explorer 11.

Even Concept IDE and Gyro may be run in browser:

You can even try some applications. For example, a simple application for converting websites to images (png) can be found here (about 100 lines of code).

Even audio codecs are supported in browser (Speex and Opus). A simple VoIP + Video application can be found here. It has about 350 lines of code.

GyroGears will be optimized next, to generate web-friendly applications (for now the applications look like classic desktop applications).

Concept 4.0

Two months ago I've decided to start planning for Concept 4.0, just after CAS 3.0 was updated to most of my users. I skipped any intermediated version because is an entire new server and client. The main feature of the CAS 4 is the scalability. For this, the new server is written entirely in Concept (dropping the C/C++ server), and is a single-thread, polling server. It relays on green threads and non-blocking IO for performance.

Here you can see  a memory usage comparison between Concept 3.0 (see the green line) and 4.0 (the blue line) for a small (mobile) application.

For an enterprise application (test was done for a real-world ERP application):

This means that a concept:// application has a very small footprint for each additional user. This is because now, you can choose to put all the users in the same process, sharing the same resources.

You could easily keep tens of thousands (possibly hundreds of thousands) of users on the same server.

This is only half of what CAS 4.0 brings. Concept 4.0 doesn't need a dedicated thin client anymore (you can still use it, but is optional). The new concept:// client is written in JavaScript and is based on web sockets and HTML5.

You can even do real-time communication applications:

Audio and video is supported directly in your browser (as long as it is Firefox, Chrome or Opera). All major browsers are running concept:// client, but Safari and Internet Explorer 11 still don't support microphone and webcam capture.

Now every GyroGears applications runs directly in browser with a very small footprint, being able to handle large amounts of simultaneous users.

I hope to have the first BETA version early January 2015.

10 years of Concept

Already 10 years since the first Concept 1.0 release... Countless white nights all for a passion. Is now the most complex project I've been working on. It was fun, sometimes frustrating, and fueled only by my passion. The Concept Book is now almost ready (I hope to find an editor in august).

Concept Client 3.0 is now finally released but available only on Windows. I hope that next week it will be available on Linux and Mac (64bit only).

Between tens of bug-fixes, I managed to add a nice feature to GyroGears: a form designer. Gyro automatically generates all the forms, but in some cases, a manual design may be needed.

 Activating the marked link will open the form designer:

This is very useful for specific layout requests. However, I don't recommend using this feature, because it could provide an inconsistent layout through the application. GyroGears forms all identical in layout for a given application. When adding a custom form, it may interfere with the application usability.

Concept 3.0 developing tools

Concept 3.0 is now finally available. The server itself is stable (the engine is available since version 2.6), but now the also Concept Client 3.0 is available.

It comes with a brand new IDE, Concept IDE 2.0.

Glade was dropped in favor of RDesigner, a Concept-native UI designer that can handle both glade and the new Concept UI XML format.

I've started a face-lift for GyroGears. It will also include a "Custom form designer" based on RDesigner that will allow the developer to define specific forms when needed, replacing the gyro auto-generated forms.

The final release is planned on the 10th birthday of Concept, in about two weeks. It will include the Concept Client 3.0 deb package and OS X image.

Concept 3.0

A lot has happened since the last time I've posted. To start with, I finally written the Concept manual, a 600 pages book describing the Concept programming language and framework. Now I'm looking for an editor that I could afford. I hope that the book will be available in October 2014.

Let's start with the new stuff in Concept 3.0

• it comes with in .deb packages for i386 and amd64
• added support for GPU programming
• added support for libvirt
• added code profiler
• added SSH APIs
• added Modbus protocol support
• added tls/ssl support via OpenSSL
• added support for the netflow protocol
• added new compiler and run-time optimizations that makes Concept run up to two times faster
• added support for green threads (beside native threads). Note that I prefer calling green threads "cheap" threads.
• has a brand new client, Concept Client 3.0 that dropped GTK in favor of QT (see bellow)

Lets start with the GPU programming. It uses OpenCL and it automatically converts Concept bytecode to GPU bytecode, for example:

As you see, all the code is written in Concept (no OpenCL code). The Parallel framework may use both concept code and/or OpenCL code. As a note, for Concept functions, all parameters must be declared by type (see foo4 definition).


Now, the Profiler worked miracles for me, making Gyro faster than ever.

It will give you real-time information directly in the application (via ProfilerForm), or you could  use the low-level APIs for background analytic:

Or even simpler, you could just choose "Profiler project" from the Concept IDE.

The green threads are a really cool toy:

These threads are scheduled directly by the Concept Core, not needing semaphores or any kind of synchronization. These threads are really fast, but are limited to one CPU core. You could combine native threads with green threads for maximum performance. Also, green threads run with the regular Concept Core (not needing the multi-threaded core).

The Concept Client was completely rewritten. I have hesitated in forcing the Gtk3 version due to stability problems on MS Windows. I've watched the Qt community, forums and list, and I was impressed by the passion of the developers, unlike the Gtk groups where there were mixed posts, both by professionals and "political" individuals involved in ego wars. So, I've rewritten Concept Client on Qt, I even made a converter from .glade files to Qt code. I like that Qt looks and performs better, supports a lot more platforms, and seems stabler. The only advantages of Gtk are the significantly lower memory usage (for the same test application, Concept Client Qt uses 102 MB of memory while Gtk 63 MB). Since I'm talking of a "Client" 40 MB difference is relatively low comparing to the advantages. Qt is faster, more responsive and behaves consistently on every platform. I find Gtk slightly more flexible on "exotic" operations, for example, adding custom controls in a tab header.

Perhaps the greatest advantage of Qt is the completeness of the framework. It includes webkit, audio, network and event webcam APIs, elements missing in Gtk (I've compiled webkit for gtk, and added OpenCV for webcam support). It also has a sheet control, that Gtk supported only with the additional GtkExtra package. However the GtkExtra developers don't seem to hurry the port to Gtk3.

It actually looks better:

I hope I will manage to post more often in the next month. In August this year, Concept will be 10 years old!

9 years and one month of Concept Application Server

It's been 9 years since I've started this project. Now is mature, tested in high load environments, over a hundred end-user solutions implemented by myself. There's a lot of new stuff, both in Concept Application Server and GyroGears.

For start, Concept 3.0 is almost here, with lots of new features. The main star is the new Concept Client for iOS.

 

For Gyro, lots of bug fixes and a new UI model for child entities was added. Now, you can show all the info using the same window (without opening new child windows). It's a tab-bar style view.

Also, drag-and-drop of files/images is now cleaner.

Mobile applications generated by GyroGears are now read/write, and use the same source code and binary for both Android and iOS.

DRM API's are available in beta, as a new feature in Concept 3.0.

Concept Client for OS X 10.8 is now up-to-date, and soon a binary packet for Concept Application Server will be available for OS X.

Concept 2.4

Concept 2.4 now available for download. As the main feature is the new JIT (sljit), and some basic code optimization (like dead-code elimination).

It almost tripled its speed since the first version with JIT. Here is an example for a basic benchmark for arrays + basic operations (lower the better):

About 6 times faster than PHP 5.4 and about the same speed with C (gcc, no optimizations).

I've also ran a Sieve benchmark (higher, the better):

The slight advantage for gcc comes from the fact that Concept core must call a function for accessing array elements, instead of a direct access (like in C/C++).

For now is fast enough for Concept to be a reasonable choice for CPU-intensive computations.

Average speed optimization for Concept/JIT is about 800%, but in some particular cases it is more than 2000%.

Concept Application Server 2.4 with 3.0 core

I've made a few tests with various programming and scripting languages (C/C++, IonMonkey, Chrome V8, Java, PHP 5.4). I've used an implementation for the sieve of Eratosthenes algorithm for the benchmark. As an unexpected result, IonMonkey (Firefox 18 javascript engine) was insanely fast (around C speed).

As a result, Concept now has a JIT compiler, that fully compiles a function upon the second run. For the standard benchmark (strings + arrays + arithmetic operations + objects) some impressive improvements were noticed. For example, arithmetic operations are about 2000% faster. Function call (context create and free) about 50% faster. Array operations are slightly faster for numbers, the same for strings. String operations already optimized remain the same. Object method call (including constructor call) about 25% faster by reusing memory (instead of freeing the memory, is conserved for the next call, same with objects). So, for the Sieve of Erathostenes repeated 900 times, these are the results (lower, the better):

As you can see, the benefits for using JIT are massive. I've chosen to use an existing JIT (Stack-less JIT compiler). It's incredibly simple and clean, and it took me about 2 days to link it to the Concept Core.
As a note, for PHP, if the Sieve interval is extended from 8193 to 1,000,000 primes, the execution time raises exponentially (v5.4 about 580 seconds, Concept with JIT about 30 seconds). A low interval was chosen as representative for real-life situations.

For GyroGears, bug fixes and a new feature: import from CSV for related members and export to Excel.

For now, Concept 2.4 with JIT compiler is available for download.