SonarQube 数据清理,从100G 到9G

Stella981
• 阅读 886

背景描述

SonarQube 自去年使用开始,已运行一年有余
它上面的 Project 数量已超过 1000
因为每个应用的每个 git 分支的每次 push 事件都会触发 Sonar 分析任务
这样会有很多的数据都存储下来,从而导致数据量很大
目前 Mysql 数据库占用磁盘空间达到100G
并且随着时间推移会继续增加
于是在想是否可以进行数据清理,以降低所使用的磁盘空间

如何进行数据清理?

那么如何进行数据清理呢?
经过不断探索与实践,形成如下解决方案:
1、调小【数据库清理器】相关参数
2、设置 master 为长期分支,并修改【保留短期分支】参数
3、清理非主分支的长期分支
4、MySQL Innodb DATA_FREE 清理

调小【数据库清理器】相关参数

配置—>通用配置 —> 数据库清理器
调小【数据库清理器】相关参数,缩短数据保留时间,以便尽早释放空间

SonarQube 数据清理,从100G 到9G

SonarQube 数据清理,从100G 到9G

设置 master 为长期分支

配置—>通用配置 —> Branches
默认情况下,名称以 “branch” 或 “release” 开头的分支将被视为长期分支,长期分支系统不会做自动清理。
将 master 设置为长期分支,那么其它分支就是短期分支,短期分支系统会做自动清理。
当然已经存在的长期分支不会自动更改为短期分支,需要先删除它,然后重新在这个分支执行 sonar 分析任务,那么它就会变为短期分支
此外,修改【清除不活跃的短分支前的保留天数】参数,改为10

SonarQube 数据清理,从100G 到9G

使用长期分支模式后,每个 Project 分支页面如下:

短期分支会显示和长期分支有差异的 issue

SonarQube 数据清理,从100G 到9G

清理非主分支的长期分支

修改为长期分支模式的两个月后,清理非主分支的长期分支
这里使用了 SonarQube API 进行了批量清理,清理任务花费了「三个多小时」~
相关 python 脚本可以参考:https://github.com/donhui/python-sonarqube/blob/master/sonarqube_branches_clean.py

MySQL Innodb DATA_FREE 清理

对 SonarQube 数据库进行分析:

mysql> SELECT table_schema "DB Name", Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB" FROM information_schema.tables GROUP BY table_schema;
+--------------------------+---------------+
| DB Name                  | DB Size in MB |
+--------------------------+---------------+
| dingtalk_develop_members |           0.1 |
| information_schema       |           0.2 |
| sonar                    |       90477.8 |
| sonar_statistics         |           5.1 |
+--------------------------+---------------+
4 rows in set (0.30 sec)

mysql> SELECT table_name AS `Table`, round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB` FROM information_schema.TABLES WHERE table_schema = 'sonar';
+--------------------------+------------+
| Table                    | Size in MB |
+--------------------------+------------+
| active_rule_parameters   |       0.03 |
| active_rules             |       0.42 |
| analysis_properties      |       0.02 |
| ce_activity              |      32.55 |
| ce_queue                 |       0.05 |
| ce_scanner_context       |     619.05 |
| ce_task_characteristics  |      51.19 |
| ce_task_input            |       1.52 |
| default_qprofiles        |       0.03 |
| duplications_index       |       0.05 |
| es_queue                 |       0.03 |
| events                   |       6.20 |
| file_sources             |   33973.41 |
| group_roles              |       2.53 |
| groups                   |       0.02 |
| groups_users             |       0.23 |
| internal_properties      |       0.02 |
| issue_changes            |     160.84 |
| issues                   |     372.92 |
| loaded_templates         |       0.03 |
| manual_measures          |       0.02 |
| metrics                  |       0.08 |
| notifications            |       0.02 |
| org_qprofiles            |       0.05 |
| organization_members     |       0.05 |
| organizations            |       0.03 |
| perm_templates_groups    |       0.02 |
| perm_templates_users     |       0.02 |
| perm_tpl_characteristics |       0.02 |
| permission_templates     |       0.02 |
| plugins                  |       0.03 |
| project_branches         |       1.80 |
| project_links            |       4.52 |
| project_measures         |   49670.27 |
| project_qprofiles        |       0.02 |
| projects                 |    5483.33 |
| properties               |       0.03 |
| qprofile_changes         |       1.92 |
| qprofile_edit_groups     |       0.02 |
| qprofile_edit_users      |       0.02 |
| quality_gate_conditions  |       0.02 |
| quality_gates            |       0.02 |
| rule_repositories        |       0.02 |
| rules                    |       3.70 |
| rules_metadata           |       0.02 |
| rules_parameters         |       0.08 |
| rules_profiles           |       0.03 |
| schema_migrations        |       0.02 |
| snapshots                |       7.69 |
| user_roles               |       0.78 |
| user_tokens              |       0.05 |
| users                    |       0.14 |
| webhook_deliveries       |       0.05 |
+--------------------------+------------+
53 rows in set (0.02 sec)

数据量大的表:

| ce_scanner_context       |     619.05 |
| file_sources             |   33973.41 |
| issue_changes            |     160.84 |
| issues                   |     372.92 |
| project_measures         |   49670.27 |
| projects                 |    5483.33 |

清理非主分支的长期分支后,发现 data_length + index_length 下降,但是磁盘空间并没有减少
向 DBA 求助,得到的解决方案为:清理数据库 DATA_FREE,回收表空间

查看表相关的元数据信息,包括 TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE ,SQL 语句及输出如下:

mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar';
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
| TABLE_NAME               | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE   | ENGINE |
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
| active_rule_parameters   | Dynamic    |        138 |       16384 |        16384 |               0 |           0 | InnoDB |
| active_rules             | Dynamic    |       4385 |      294912 |       147456 |               0 |           0 | InnoDB |
| analysis_properties      | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| ce_activity              | Dynamic    |      74861 |    16793600 |     17334272 |               0 |     4194304 | InnoDB |
| ce_queue                 | Dynamic    |          0 |       16384 |        32768 |               0 |           0 | InnoDB |
| ce_scanner_context       | Dynamic    |      17122 |   649117696 |            0 |               0 |  1286602752 | InnoDB |
| ce_task_characteristics  | Dynamic    |     228069 |    36290560 |     17383424 |               0 |     5242880 | InnoDB |
| ce_task_input            | Dynamic    |          0 |      344064 |            0 |               0 |   113246208 | InnoDB |
| default_qprofiles        | Dynamic    |         14 |       16384 |        16384 |               0 |           0 | InnoDB |
| duplications_index       | Dynamic    |          0 |       16384 |        32768 |               0 |           0 | InnoDB |
| es_queue                 | Dynamic    |          0 |       16384 |        16384 |               0 |     5242880 | InnoDB |
| events                   | Dynamic    |       2882 |     3342336 |      2457600 |               0 |     5242880 | InnoDB |
| file_sources             | Dynamic    |     168412 |  8828141568 |    281624576 |               0 | 33689698304 | InnoDB |
| group_roles              | Dynamic    |       7000 |      507904 |      2146304 |               0 |     6291456 | InnoDB |
| groups                   | Dynamic    |         48 |       16384 |            0 |               0 |           0 | InnoDB |
| groups_users             | Dynamic    |       1060 |       81920 |       163840 |               0 |           0 | InnoDB |
| internal_properties      | Dynamic    |          3 |       16384 |            0 |               0 |           0 | InnoDB |
| issue_changes            | Dynamic    |     762484 |   104284160 |     64372736 |               0 |     6291456 | InnoDB |
| issues                   | Dynamic    |     184354 |   200441856 |    164954112 |               0 |    35651584 | InnoDB |
| loaded_templates         | Dynamic    |          2 |       16384 |        16384 |               0 |           0 | InnoDB |
| manual_measures          | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| metrics                  | Dynamic    |        169 |       65536 |        16384 |               0 |           0 | InnoDB |
| notifications            | Dynamic    |          0 |       16384 |            0 |               0 |     6291456 | InnoDB |
| org_qprofiles            | Dynamic    |         34 |       16384 |        32768 |               0 |           0 | InnoDB |
| organization_members     | Dynamic    |        393 |       49152 |            0 |               0 |           0 | InnoDB |
| organizations            | Dynamic    |          1 |       16384 |        16384 |               0 |           0 | InnoDB |
| perm_templates_groups    | Dynamic    |          4 |       16384 |            0 |               0 |           0 | InnoDB |
| perm_templates_users     | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| perm_tpl_characteristics | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| permission_templates     | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| plugins                  | Dynamic    |         23 |       16384 |        16384 |               0 |           0 | InnoDB |
| project_branches         | Dynamic    |       1673 |      229376 |       114688 |               0 |           0 | InnoDB |
| project_links            | Dynamic    |      26829 |     4734976 |            0 |               0 |     4194304 | InnoDB |
| project_measures         | Dynamic    |   14545516 |  7061143552 |  10892787712 |               0 | 36237737984 | InnoDB |
| project_qprofiles        | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| projects                 | Dynamic    |     398678 |  1495941120 |   1684144128 |               0 |  2999975936 | InnoDB |
| properties               | Dynamic    |        125 |       16384 |        16384 |               0 |           0 | InnoDB |
| qprofile_changes         | Dynamic    |       5049 |     1589248 |       425984 |               0 |     4194304 | InnoDB |
| qprofile_edit_groups     | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| qprofile_edit_users      | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| quality_gate_conditions  | Dynamic    |         12 |       16384 |            0 |               0 |           0 | InnoDB |
| quality_gates            | Dynamic    |          0 |       16384 |            0 |               0 |           0 | InnoDB |
| rule_repositories        | Dynamic    |         37 |       16384 |            0 |               0 |           0 | InnoDB |
| rules                    | Dynamic    |       3185 |     3686400 |       196608 |               0 |     4194304 | InnoDB |
| rules_metadata           | Dynamic    |          5 |       16384 |            0 |               0 |           0 | InnoDB |
| rules_parameters         | Dynamic    |        328 |       65536 |        16384 |               0 |           0 | InnoDB |
| rules_profiles           | Dynamic    |         33 |       16384 |        16384 |               0 |           0 | InnoDB |
| schema_migrations        | Dynamic    |        237 |       16384 |            0 |               0 |           0 | InnoDB |
| snapshots                | Dynamic    |       4715 |     1589248 |       327680 |               0 |     2097152 | InnoDB |
| user_roles               | Dynamic    |       6338 |      507904 |       311296 |               0 |           0 | InnoDB |
| user_tokens              | Dynamic    |         34 |       16384 |        32768 |               0 |           0 | InnoDB |
| users                    | Dynamic    |        438 |      114688 |        32768 |               0 |           0 | InnoDB |
| webhook_deliveries       | Dynamic    |          0 |       16384 |        32768 |               0 |           0 | InnoDB |
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
53 rows in set (0.00 sec)

数据库清理 DATA_FREE,回收表空间,SQL 语句及输出如下:

mysql> ALTER TABLE file_sources ENGINE=InnoDB;
Query OK, 0 rows affected (2 min 41.44 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE issue_changes ENGINE=InnoDB;
Query OK, 0 rows affected (4.39 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE ce_scanner_context ENGINE=InnoDB;
Query OK, 0 rows affected (14.82 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE project_measures ENGINE=InnoDB;
Query OK, 0 rows affected (3 min 55.74 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE projects ENGINE=InnoDB;
Query OK, 0 rows affected (25.27 sec)
Records: 0  Duplicates: 0  Warnings: 0

清理 DATA_FREE 之后,再次查看表相关的元数据信息:

mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar';
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
| TABLE_NAME               | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE | ENGINE |
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
| active_rule_parameters   | Dynamic    |        138 |       16384 |        16384 |               0 |         0 | InnoDB |
| active_rules             | Dynamic    |       4385 |      294912 |       147456 |               0 |         0 | InnoDB |
| analysis_properties      | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| ce_activity              | Dynamic    |      74971 |    16793600 |     17334272 |               0 |   4194304 | InnoDB |
| ce_queue                 | Dynamic    |          0 |       16384 |        32768 |               0 |         0 | InnoDB |
| ce_scanner_context       | Dynamic    |      17451 |   659079168 |            0 |               0 |   6291456 | InnoDB |
| ce_task_characteristics  | Dynamic    |     228289 |    36290560 |     17383424 |               0 |   5242880 | InnoDB |
| ce_task_input            | Dynamic    |          0 |     1589248 |            0 |               0 | 113246208 | InnoDB |
| default_qprofiles        | Dynamic    |         14 |       16384 |        16384 |               0 |         0 | InnoDB |
| duplications_index       | Dynamic    |          0 |       16384 |        32768 |               0 |         0 | InnoDB |
| es_queue                 | Dynamic    |          0 |       16384 |        16384 |               0 |   5242880 | InnoDB |
| events                   | Dynamic    |       3300 |     3342336 |      2457600 |               0 |   5242880 | InnoDB |
| file_sources             | Dynamic    |     122205 |  3011493888 |     22626304 |               0 |   5242880 | InnoDB |
| group_roles              | Dynamic    |       7000 |      507904 |      2146304 |               0 |   6291456 | InnoDB |
| groups                   | Dynamic    |         48 |       16384 |            0 |               0 |         0 | InnoDB |
| groups_users             | Dynamic    |       1060 |       81920 |       163840 |               0 |         0 | InnoDB |
| internal_properties      | Dynamic    |          3 |       16384 |            0 |               0 |         0 | InnoDB |
| issue_changes            | Dynamic    |     731182 |   101318656 |     43089920 |               0 |   6291456 | InnoDB |
| issues                   | Dynamic    |     184366 |   200441856 |    164954112 |               0 |  35651584 | InnoDB |
| loaded_templates         | Dynamic    |          2 |       16384 |        16384 |               0 |         0 | InnoDB |
| manual_measures          | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| metrics                  | Dynamic    |        169 |       65536 |        16384 |               0 |         0 | InnoDB |
| notifications            | Dynamic    |          0 |       16384 |            0 |               0 |   6291456 | InnoDB |
| org_qprofiles            | Dynamic    |         34 |       16384 |        32768 |               0 |         0 | InnoDB |
| organization_members     | Dynamic    |        393 |       49152 |            0 |               0 |         0 | InnoDB |
| organizations            | Dynamic    |          1 |       16384 |        16384 |               0 |         0 | InnoDB |
| perm_templates_groups    | Dynamic    |          4 |       16384 |            0 |               0 |         0 | InnoDB |
| perm_templates_users     | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| perm_tpl_characteristics | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| permission_templates     | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| plugins                  | Dynamic    |         23 |       16384 |        16384 |               0 |         0 | InnoDB |
| project_branches         | Dynamic    |       1686 |      229376 |       114688 |               0 |         0 | InnoDB |
| project_links            | Dynamic    |      26866 |     4734976 |            0 |               0 |   4194304 | InnoDB |
| project_measures         | Dynamic    |   10381786 |  2282733568 |   1306165248 |               0 |  51380224 | InnoDB |
| project_qprofiles        | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| projects                 | Dynamic    |     374448 |   314392576 |    132235264 |               0 |   4194304 | InnoDB |
| properties               | Dynamic    |        125 |       16384 |        16384 |               0 |         0 | InnoDB |
| qprofile_changes         | Dynamic    |       5049 |     1589248 |       425984 |               0 |   4194304 | InnoDB |
| qprofile_edit_groups     | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| qprofile_edit_users      | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| quality_gate_conditions  | Dynamic    |         12 |       16384 |            0 |               0 |         0 | InnoDB |
| quality_gates            | Dynamic    |          0 |       16384 |            0 |               0 |         0 | InnoDB |
| rule_repositories        | Dynamic    |         37 |       16384 |            0 |               0 |         0 | InnoDB |
| rules                    | Dynamic    |       3185 |     3686400 |       196608 |               0 |   4194304 | InnoDB |
| rules_metadata           | Dynamic    |          5 |       16384 |            0 |               0 |         0 | InnoDB |
| rules_parameters         | Dynamic    |        328 |       65536 |        16384 |               0 |         0 | InnoDB |
| rules_profiles           | Dynamic    |         33 |       16384 |        16384 |               0 |         0 | InnoDB |
| schema_migrations        | Dynamic    |        237 |       16384 |            0 |               0 |         0 | InnoDB |
| snapshots                | Dynamic    |       4737 |     1589248 |       327680 |               0 |   4194304 | InnoDB |
| user_roles               | Dynamic    |       6338 |      507904 |       311296 |               0 |         0 | InnoDB |
| user_tokens              | Dynamic    |         34 |       16384 |        32768 |               0 |         0 | InnoDB |
| users                    | Dynamic    |        438 |      114688 |        32768 |               0 |         0 | InnoDB |
| webhook_deliveries       | Dynamic    |          0 |       16384 |        32768 |               0 |         0 | InnoDB |
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
53 rows in set (0.06 sec)

从上面可以看出,相关表(如:project_measures,projects,file_sources)的 DATA_FREE 的大小有很大幅度下降。
然后,查看数据库所占磁盘空间,也有大幅度下降:100G —> 9G 。

至此,SonarQube 的数据清理已取得很好的成果,以此为记。

点赞
收藏
评论区
推荐文章
blmius blmius
3年前
MySQL:[Err] 1292 - Incorrect datetime value: ‘0000-00-00 00:00:00‘ for column ‘CREATE_TIME‘ at row 1
文章目录问题用navicat导入数据时,报错:原因这是因为当前的MySQL不支持datetime为0的情况。解决修改sql\mode:sql\mode:SQLMode定义了MySQL应支持的SQL语法、数据校验等,这样可以更容易地在不同的环境中使用MySQL。全局s
皕杰报表之UUID
​在我们用皕杰报表工具设计填报报表时,如何在新增行里自动增加id呢?能新增整数排序id吗?目前可以在新增行里自动增加id,但只能用uuid函数增加UUID编码,不能新增整数排序id。uuid函数说明:获取一个UUID,可以在填报表中用来创建数据ID语法:uuid()或uuid(sep)参数说明:sep布尔值,生成的uuid中是否包含分隔符'',缺省为
待兔 待兔
3个月前
手写Java HashMap源码
HashMap的使用教程HashMap的使用教程HashMap的使用教程HashMap的使用教程HashMap的使用教程22
Jacquelyn38 Jacquelyn38
3年前
2020年前端实用代码段,为你的工作保驾护航
有空的时候,自己总结了几个代码段,在开发中也经常使用,谢谢。1、使用解构获取json数据let jsonData  id: 1,status: "OK",data: 'a', 'b';let  id, status, data: number   jsonData;console.log(id, status, number )
Easter79 Easter79
3年前
Twitter的分布式自增ID算法snowflake (Java版)
概述分布式系统中,有一些需要使用全局唯一ID的场景,这种时候为了防止ID冲突可以使用36位的UUID,但是UUID有一些缺点,首先他相对比较长,另外UUID一般是无序的。有些时候我们希望能使用一种简单一些的ID,并且希望ID能够按照时间有序生成。而twitter的snowflake解决了这种需求,最初Twitter把存储系统从MySQL迁移
Wesley13 Wesley13
3年前
mysql设置时区
mysql设置时区mysql\_query("SETtime\_zone'8:00'")ordie('时区设置失败,请联系管理员!');中国在东8区所以加8方法二:selectcount(user\_id)asdevice,CONVERT\_TZ(FROM\_UNIXTIME(reg\_time),'08:00','0
Wesley13 Wesley13
3年前
00:Java简单了解
浅谈Java之概述Java是SUN(StanfordUniversityNetwork),斯坦福大学网络公司)1995年推出的一门高级编程语言。Java是一种面向Internet的编程语言。随着Java技术在web方面的不断成熟,已经成为Web应用程序的首选开发语言。Java是简单易学,完全面向对象,安全可靠,与平台无关的编程语言。
Wesley13 Wesley13
3年前
MySQL部分从库上面因为大量的临时表tmp_table造成慢查询
背景描述Time:20190124T00:08:14.70572408:00User@Host:@Id:Schema:sentrymetaLast_errno:0Killed:0Query_time:0.315758Lock_
为什么mysql不推荐使用雪花ID作为主键
作者:毛辰飞背景在mysql中设计表的时候,mysql官方推荐不要使用uuid或者不连续不重复的雪花id(long形且唯一),而是推荐连续自增的主键id,官方的推荐是auto_increment,那么为什么不建议采用uuid,使用uuid究
Python进阶者 Python进阶者
9个月前
Excel中这日期老是出来00:00:00,怎么用Pandas把这个去除
大家好,我是皮皮。一、前言前几天在Python白银交流群【上海新年人】问了一个Pandas数据筛选的问题。问题如下:这日期老是出来00:00:00,怎么把这个去除。二、实现过程后来【论草莓如何成为冻干莓】给了一个思路和代码如下:pd.toexcel之前把这