The experiment is related to a real problem, where messages comes from remote systems, providing information about these systems. The receiving system gets knowledge about remote systems and constructs current state knowledge, together with aggregated state and a history of changes of these systems.
The implementation is used to learn Go lang and to improve parallel developing skills.
Architecture
The incomming data is provided through RabbitMQ query. Expected results are
- the current (live) state, of the remote system(s).
- aggregated state of the remote system(s), for a given period of time.
- history data of the remote system(s).
In the implemented system, the live state and aggregated state are stored in a shared memory and used in a calculations module. The history data represents state changes in the remote system(s) and is stored to the database after is applied to the live state and aggregated state.
The DataCalculator routine proces the incomming data and applies to the main state; then it sends the data to the history synchronyzer through a Go channel, which then writes data to the database.
Aggregated state synchronyzer and live state synchronyzer reacts to a ticker signals and stores changed data from the state to the database.
The aggregated state synchronyzer sends ACK signal to a rabbit queue after the data is stored to database by marking that the incomming messages are fully processed, including all calculations and a storage to database. In case the system was stopped in the middle of calculations, all RabbitMQ messages would be returned to the queue, if they were not stored to the database.
When the system is restarted, the messages will be used again to calculate aggregated state at the moment it was interrupted.
The Observer has access to the main state and all channels to show statistics about a system in a HTTP API response. A js script calls the observer API endpoint each 2 seconds and displays statistics in a html page.
All blue ellipses in the schema represents a separate module running in a Go routine.
The Shared memory state is a shared data holder between all modules. Every access is proteced from conflicts of concurrent access by using Go Mutex'es.
While the state data is written to a database by Aggregated state synchronizer and Live state synchronizer, the state is updated by the DataCalculator in parallel; to avoid update conflicts, synchronizer modules makes snapshots (copies of the data from the main state) before writing changes to the database.
Challenges with the database
When the history data is written to the database, each record corresponds several different tables in the database, related to each other through auto generated ids.
In the simplest, but non-effective way, after each record written to the database, the repository must obtain generated id, and assign it to the related object before sending another write query to the database. This is not effective in two ways:
- Records must be written one-by-one.
- The repository must wait for the response of the generated id before making a write of the related objects.
So the solution is to use uuid, generated in the application code and assigning same uuid to related records. Later the auto genrated ids may be assigned by running simple update query inside database, which finds related records between two tables and makes id assignment to the related column.
Statistics
In the presented solution, test was run with a producer which immitates 10000 remote systems, generating 1000 new messages per second. When the Rabbit queue fills to several tens of thousand messages, the system with all modules is starded, and the DataCalculator starts draining a queue and applying all required calculations by about 1200 messages per second.
- Memory usage is 1288M for the whole system, which is almost unchanging during running time; after an hour of the experiment Memory usage is 1289M.
- Processor usage of the system is about 20%
- Database processor usage is about 60%
- The machine where experiment was run has Intel Core i7 processor with 20 cores: 12th Gen Intel® Core™ i7-12700H × 20
Conclusions
- The system consumes messages and makes calculations in 1200 messages per second.
- The database writes runs in separate Go routines, providing an excellent example of Go lang suitability for solving such problems.
- The implemented solution does not leak memory and does not overwhelm processors.
- By doing this project I learned Go parallel development principles in a practical way.
2026-05-22