Scaling MySQL database, increasing performance to many connections

Question

I have a monitoring system where my customers can register their terminals and his terminals send a periodically (5min) keepalive signal to my website to inform that it is online. customers also can access a monitoring page that show all his terminals and update it's status using ajax in an interval of 20sec.

Plus information: a terminal is a android device, customer have to install an app from google play.

THE PROBLEM IS: With increasing customer number, many peoples access the monitoring page at the same time that is almost flooding server with many requests, and on the other side. each time more terminals is comming and flooding more with it's keepalive signal. so I have besides the common pages (login, many CRUDs etc) dozens phisical terminals sending keepalive signal through internet flooding my database, and many users accessing monitoring pages to get informed their terminals are online. it seems like a time bomb. because I don't know if mysql will support when number of terminals reach hundreds and counting.

PLUS we're already noting our server is decreasing performance along the time it is running. We restart it, and it's very fast, but along the time, it will lose performance

SOLUTION

• What can I do to improve performance or make the model more scalable? there is an design pattern for this kind of monitoring system that is more scalable?

• There is any gain if I separate two mysql databases, one for common use (access pages, cruds etc) and another for monitoring system?

• There is any gain to use MongoDB just for the monitoring part of the system?

additional information:

mysql Ver 14.14 Distrib 5.5.43, for Linux (x86_64) using readline 5.1

PHP 5.4.40 (cli) (built: Apr 15 2015 15:55:28)

Jetty 8.1.14 (for java server side that comunicates with android app)

Server Mon

Free memory ........: 17.84 Gb

Total memory........: 20 Gb

Used memory.........: 2.16 Gb

RAM.................: 20 Kb

JVM Free memory.....: 1.56 Gb

JVM Maximum memory..: 3.93 Gb

JVM Total available.: 1.93 Gb

**************************************

Total (cores).: 10

CPU idle......: 4.9%

CPU nice......: 0.0%

CPU system....: 4183000.0%

CPU total.....: 5.0%

CPU user......: 2.6%

**************************************

Total space (bytes)..: 600 Gb

Free space (bytes)...: 595.64 Gb

Usable space (bytes).: 595.64 Gb

PART OF MODEL AND MONITORING PAGE'S QUERY

This is terminals table

    CREATE TABLE IF NOT EXISTS `GM_PLAYER` (
      `ID_PLAYER` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
      `DS_GCM_ID` VARCHAR(250) NULL,
      `DT_CRIACAO` DATETIME NOT NULL,
      `DS_PLAYER` VARCHAR(100) NOT NULL,
      `DS_JANELA_HEIGHT` INT(11) NOT NULL DEFAULT '1024',
      `DS_JANELA_WIDTH` INT(11) NOT NULL DEFAULT '768',
      `DS_JANELA_POS_X` INT(11) NOT NULL DEFAULT '0',
      `DS_JANELA_POS_Y` INT(11) NOT NULL DEFAULT '0',
      `DS_WALLPAPER` VARCHAR(255) NULL DEFAULT NULL,
      `FL_ATIVO` CHAR(1) NOT NULL DEFAULT 'N',
      `FL_FULL_SCREEN` CHAR(1) NOT NULL DEFAULT 'S',
      `FL_MOUSE_VISIBLE` CHAR(1) NOT NULL DEFAULT 'N',
      `DS_SERIAL` VARCHAR(50) NULL DEFAULT NULL,
      `VERSAO_APP` VARCHAR(20) NULL DEFAULT NULL,
      `VERSAO_OS` VARCHAR(20) NULL DEFAULT NULL,
      `FL_EXIBIR_STATUS_BAR` CHAR(1) NOT NULL DEFAULT 'S',
      `ID_GRADE_PROGRAMACAO` BIGINT UNSIGNED NULL DEFAULT NULL,
      `ID_CLIENTE` BIGINT UNSIGNED NULL,
      `ID_PONTO` BIGINT UNSIGNED NULL,
      `FL_ATIVO_SISTEMA` CHAR(1) NOT NULL DEFAULT 'S',
      `FL_DEBUG` CHAR(1) NOT NULL DEFAULT 'N',
      `VERSAO_APP_UPDATE` VARCHAR(20) NULL,
      `FL_ESTADO_MONITOR` CHAR(1) NOT NULL DEFAULT 'L',
      `FL_DEVICE_ROOTED` CHAR(1) DEFAULT 'N',
      `DT_ATIVACAO` DATETIME ,
      `DT_EXPIRA` DATETIME ,
      `FL_EXCLUIDO` CHAR(1) DEFAULT 'N' ,
      `ID_USUARIO` BIGINT UNSIGNED NOT NULL,
      `ID_PACOTE` BIGINT UNSIGNED ,

      `DS_IMG_BARRA` VARCHAR(255),
      `FL_EXIBIR_HORA` CHAR(1),
      `DS_TEXTO_BARRA` TEXT,

      PRIMARY KEY (`ID_PLAYER`),
      UNIQUE INDEX `UQ_GM_PLAYER_ID_PLAYER` (`ID_PLAYER` ASC),
      INDEX `ID_GRADE_PROGRAMACAO` (`ID_GRADE_PROGRAMACAO` ASC),
      INDEX `FK_GM_PLAYER_GM_CLIENTE_idx` (`ID_CLIENTE` ASC),
      CONSTRAINT `FK_GM_PLAYER_GM_USUARIO` FOREIGN KEY (`ID_USUARIO`) REFERENCES `GM_USUARIO` (`ID_USUARIO`) ON DELETE RESTRICT,
      CONSTRAINT `FK_GM_PLAYER_GM_GRADE_PROGRAMACAO` FOREIGN KEY (`ID_GRADE_PROGRAMACAO`) REFERENCES `GM_GRADE_PROGRAMACAO` (`ID_GRADE_PROGRAMACAO`) ON DELETE RESTRICT,
      CONSTRAINT `FK_GM_PLAYER_GM_CLIENTE` FOREIGN KEY (`ID_CLIENTE`) REFERENCES `GM_CLIENTE` (`ID_CLIENTE`) ON DELETE RESTRICT
    )
    ENGINE = InnoDB
    AUTO_INCREMENT = 5
    DEFAULT CHARACTER SET = latin1;

another used tables

    CREATE TABLE IF NOT EXISTS `GM_CLIENTE` (
      `ID_CLIENTE` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
      `DT_CRIACAO` DATETIME NOT NULL,
      `DS_CLIENTE` VARCHAR(50) NOT NULL,
      `FL_ATIVO` ENUM('S','N') NULL DEFAULT 'S',
      `ID_CONTATO` BIGINT UNSIGNED NOT NULL,
      `ID_ENDERECO` BIGINT UNSIGNED NOT NULL,
      PRIMARY KEY (`ID_CLIENTE`),
      UNIQUE INDEX `UQ_Cliente_ID_CLIENTE` (`ID_CLIENTE` ASC),
      INDEX `fk_GM_CLIENTE_GM_CONTATO1_idx` (`ID_CONTATO` ASC),
      INDEX `fk_GM_CLIENTE_GM_ENDERECO1_idx` (`ID_ENDERECO` ASC),
      CONSTRAINT `fk_GM_CLIENTE_GM_CONTATO1`
        FOREIGN KEY (`ID_CONTATO`)
        REFERENCES `GM_CONTATO` (`ID_CONTATO`)
        ON DELETE RESTRICT,
      CONSTRAINT `fk_GM_CLIENTE_GM_ENDERECO1`
        FOREIGN KEY (`ID_ENDERECO`)
        REFERENCES `GM_ENDERECO` (`ID_ENDERECO`)
        ON DELETE RESTRICT)
    ENGINE = InnoDB
    AUTO_INCREMENT = 2
    DEFAULT CHARACTER SET = latin1;

    CREATE TABLE GM_USUARIO_CLIENTE ( 
        ID_USUARIO_CLIENTE INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
        ID_CLIENTE  BIGINT UNSIGNED ,
        ID_USUARIO  BIGINT UNSIGNED
    );

This is the table where I update every time I receive a new terminal keepalive signal

    CREATE TABLE IF NOT EXISTS `GM_LOG_PLAYER` (
      `id_log_player` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
      `dt_criacao` DATETIME NOT NULL,
      `id_player` BIGINT UNSIGNED NULL,
      `qtd_midias_exibidas` INT(11) NULL,
      `id_ultima_midia_exibida` BIGINT UNSIGNED NULL,
      `up_time_android` bigint(20) unsigned default '0',
      `up_time_app` bigint(20) unsigned default '0',
      `mem_utilizada` BIGINT(20) NULL,
      `mem_disponivel` BIGINT(20) NULL,
      `hd_disponivel` BIGINT(20) NULL,
      `hd_utilizado` BIGINT(20) NULL,
      PRIMARY KEY (`id_log_player`),
      UNIQUE INDEX `UQ_id_log_player` (`id_log_player` ASC),
      INDEX `FK_GM_LOG_PLAYER_GM_PLAYER_idx` (`id_player` ASC),
      INDEX `FK_GM_LOG_PLAYER_GM_MIDIA_idx` (`id_ultima_midia_exibida` ASC),
      CONSTRAINT `FK_GM_LOG_PLAYER_GM_PLAYER`
        FOREIGN KEY (`id_player`)
        REFERENCES `GM_PLAYER` (`ID_PLAYER`)
        ON DELETE CASCADE,
      CONSTRAINT `FK_GM_LOG_PLAYER_GM_MIDIA`
        FOREIGN KEY (`id_ultima_midia_exibida`)
        REFERENCES `GM_MIDIA` (`ID_MIDIA`))
    ENGINE = InnoDB
    AUTO_INCREMENT = 3799
    DEFAULT CHARACTER SET = latin1;


    CREATE TABLE IF NOT EXISTS `GM_GRADE_PROGRAMACAO` (
      `ID_GRADE_PROGRAMACAO` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
      `DT_CRIACAO` DATETIME NOT NULL,
      `DS_GRADE_PROGRAMACAO` VARCHAR(100) NULL DEFAULT NULL,
      `ID_USUARIO` BIGINT UNSIGNED NOT NULL,
      PRIMARY KEY (`ID_GRADE_PROGRAMACAO`),
      UNIQUE INDEX `UQ_GM_GRADE_PROGRAMACAO_ID_GRADE_PROGRAMACAO` (`ID_GRADE_PROGRAMACAO` ASC),
      INDEX `fk_GM_GRADE_PROGRAMACAO_GM_USUARIO1_idx` (`ID_USUARIO` ASC),
      CONSTRAINT `fk_GM_GRADE_PROGRAMACAO_GM_USUARIO1`
        FOREIGN KEY (`ID_USUARIO`)
        REFERENCES `GM_USUARIO` (`ID_USUARIO`)
        ON DELETE RESTRICT)
    ENGINE = InnoDB
    AUTO_INCREMENT = 3
    DEFAULT CHARACTER SET = latin1;

This is the query executed periodically through ajax requests to update monitoring page

    SELECT * FROM (
        SELECT
            LOG.id_log_player ,
            LOG.dt_criacao ,
            DATE_FORMAT (LOG.DT_CRIACAO , '%d/%m/%Y %H:%i:%s') F_DT_CRIACAO ,
            (CURRENT_TIMESTAMP - LOG.DT_CRIACAO) AS IDADE_REGISTRO ,
            LOG.qtd_midias_exibidas ,
            LOG.id_ultima_midia_exibida ,
            LOG.up_time_android ,
            LOG.up_time_app ,
            LOG.mem_utilizada ,
            LOG.mem_disponivel ,
            LOG.hd_disponivel ,
            LOG.hd_utilizado ,
            PLA.FL_MONITOR_LIGADO,

            CLI.DS_CLIENTE ,

            PLA.ID_PLAYER id_player ,
            PLA.DS_PLAYER ,
            PLA.ID_CLIENTE ,
            PLA.VERSAO_APP ,
            PLA.FL_ATIVO PLA_FL_ATIVO ,
            PLA.ID_GRADE_PROGRAMACAO ,
            PLA.FL_DEVICE_ROOTED ,
            PLA.DS_GCM_ID ,
            PLA.FL_HDMI_LIGADO ,

            -- IF(PLA.FL_ATIVO='N',0,IF(PLA.ID_GRADE_PROGRAMACAO IS NULL,0,IF(PLA.ID_GRADE_PROGRAMACAO='0',0,1))) ATIVO,
            IF(PLA.FL_ATIVO='N',0,1) ATIVO,
            DATE_FORMAT (LOG.DT_CRIACAO , '%Y%m%d%H%i%s') TIME_STAMP_CRIACAO ,
            DATE_FORMAT (LOG.DT_CRIACAO , '%d/%m às %H:%i') F_DT_CRIACAO_MIN ,
            -- (CURRENT_TIMESTAMP - LOG.DT_CRIACAO) ESPERA_NOVA_COMUNICACAO ,
            --GRA.ID_GRADE_PROGRAMACAO GRA_ID_GRADE ,
            GRA.DS_GRADE_PROGRAMACAO GRA_DS_GRADE_PROGRAMACAO,
            MID.DS_PATH_THUMB THUMB_ULTMID
        FROM GM_PLAYER          PLA
        LEFT JOIN GM_CLIENTE CLI USING ( ID_CLIENTE )
        LEFT JOIN GM_USUARIO_CLIENTE GUC USING ( ID_CLIENTE )
        LEFT JOIN GM_LOG_PLAYER         LOG USING ( ID_PLAYER )
        LEFT JOIN GM_GRADE_PROGRAMACAO GRA USING ( ID_GRADE_PROGRAMACAO )
        -- LEFT JOIN GM_GRADE_PROGRAMACAO GRA ON ( PLA.ID_GRADE_PROGRAMACAO = GRA.ID_GRADE_PROGRAMACAO )
        LEFT JOIN GM_MIDIA              MID ON ( LOG.ID_ULTIMA_MIDIA_EXIBIDA = MID.ID_MIDIA )
        WHERE PLA.ID_USUARIO = ?
        AND PLA.FL_EXCLUIDO = 'N'
        AND PLA.FL_ATIVO = 'S'
        ORDER BY LOG.DT_CRIACAO DESC
    ) TBALL
    GROUP BY ID_PLAYER
    ORDER BY PLA_FL_ATIVO DESC , DT_CRIACAO DESC 

EXPLAIN QUERY ABOVE (taken from development database)

+----+-------------+------------+--------+------------------------------------------------------+----------------------------------------------+---------+--------------------------------------+-------+----------------------------------------------+
| id | select_type | table      | type   | possible_keys                                        | key                                          | key_len | ref                                  | rows  | Extra                                        |
+----+-------------+------------+--------+------------------------------------------------------+----------------------------------------------+---------+--------------------------------------+-------+----------------------------------------------+
|  1 | PRIMARY     | <derived2> | ALL    | NULL                                                 | NULL                                         | NULL    | NULL                                 | 37752 | Using temporary; Using filesort              |
|  2 | DERIVED     | PLA        | ALL    | NULL                                                 | NULL                                         | NULL    | NULL                                 |    44 | Using where; Using temporary; Using filesort |
|  2 | DERIVED     | CLI        | eq_ref | PRIMARY,UQ_Cliente_ID_CLIENTE                        | PRIMARY                                      | 8       | imidiatv.PLA.ID_CLIENTE              |     1 | NULL                                         |
|  2 | DERIVED     | GUC        | ref    | fk_GM_CLIENTE_has_GM_USUARIO_GM_CLIENTE1_idx         | fk_GM_CLIENTE_has_GM_USUARIO_GM_CLIENTE1_idx | 8       | imidiatv.PLA.ID_CLIENTE              |     1 | Using index                                  |
|  2 | DERIVED     | LOG        | ref    | FK_GM_LOG_PLAYER_GM_PLAYER_idx                       | FK_GM_LOG_PLAYER_GM_PLAYER_idx               | 9       | imidiatv.PLA.ID_PLAYER               |   858 | NULL                                         |
|  2 | DERIVED     | GRA        | eq_ref | PRIMARY,UQ_GM_GRADE_PROGRAMACAO_ID_GRADE_PROGRAMACAO | PRIMARY                                      | 8       | imidiatv.PLA.ID_GRADE_PROGRAMACAO    |     1 | NULL                                         |
|  2 | DERIVED     | MID        | eq_ref | PRIMARY,UQ_GM_MIDIA_ID_MIDIA                         | PRIMARY                                      | 8       | imidiatv.LOG.id_ultima_midia_exibida |     1 | NULL                                         |
+----+-------------+------------+--------+------------------------------------------------------+----------------------------------------------+---------+--------------------------------------+-------+----------------------------------------------+

Thanks in advance

Answer 1

Partial answer...

One aspect of scaling is to minimize the disk footprint so that caching will be more effective. Toward that end, here are some suggestions:

  PRIMARY KEY                     (`id_log_player`),
  UNIQUE INDEX `UQ_id_log_player` (`id_log_player` ASC),

A PRIMARY KEY is a UNIQUE key, so the latter is redundant and wasteful of disk space and INSERT time. DROP it.

INT is 4 bytes; BIGINT is 8 bytes. ID_xx INT UNSIGNED can handle up to 4 billion values; do you really need to go beyond 4 billion? In InnoDB, each secondary key contains a copy of the PRIMARY KEY, meaning that an unnecessary BIGINT PK consumes a lot more space.

Your tables are latin1; are you limiting the App to western languages? If you change to utf8 (or utf8mb4), I will point out wasted space for CHAR(1).

Please perform EXPLAIN SELECT ... with the tables as they stand; then make some of the changes below and do the EXPLAIN again. I'm thinking that the difference may be dramatic. I expect the part dealing with

    LEFT JOIN GM_USUARIO_CLIENTE GUC USING ( ID_CLIENTE )

to be quite 'dramatic'.

If GM_USUARIO_CLIENTE is a "many-to-many" mapping, ...

• Get rid of the AUTO_INCREMENT; instead use PRIMARY KEY(ID_CLIENTE, ID_USUARIO) to save some space and make it more efficient. (And if you do go beyond 4 billion CLIENTEs, etc, the INT would not suffice!)
• Add two indexes so that lookups will be much faster. (1) the PK (above), and (2) the other direction: INDEX(ID_USUARIO, ID_CLIENTE). Without those, JOINs involving that table will be slower and slower as you scale.

Date arithmetic is not this simple:

(CURRENT_TIMESTAMP - LOG.DT_CRIACAO)

Study the manual page on date functions; it is more complex to subtract TIMESTAMP - DATETIME. If you will be spanning timezones, be careful which datatype you use for what.

I see this pattern:

SELECT * FROM (
    SELECT ...
    ORDER BY ...    -- ??
    ) x
GROUP BY ...

What were you hoping to achieve? The optimizer is free to ignore the ORDER BY in the subquery. (Although, it may actually be performing it.)

Don't use LEFT unless you have a reason for it.

This clause

    WHERE PLA.ID_USUARIO = ?
      AND PLA.FL_EXCLUIDO = 'N'
      AND PLA.FL_ATIVO = 'S'

would benefit (greatly?) from INDEX(ID_USUARIO, FL_EXCLUIDO, FL_ATIVO). The order (in this case) of the columns in the index does not matter. If those two flags are changing frequently, do not include them in the INDEX -- UPDATEs might be slowed down more than SELECTs would benefit.

Those were the easy-to-spot suggestions. EXPLAIN may help spot more suggestions. And do you have other SELECTs?

I said "partial solution". Was that SELECT the "monitoring select"? Let's also check the periodic UPDATEs.