limits - Mysql start dropping connection when peak hours -

our database start dropping connection during peak hours. there 3% connection dropped , mysql return message "mysqli::mysqli(): (hy000/2003): can't connect mysql server on '10.88.***' (99)" . have tuned several mysql , kernel parameters, not help: -/etc/sysctl.conf:

net.ipv4.tcp_max_tw_buckets = 524288 net.core.somaxconn = 2048 net.ipv4.ip_local_port_range = 1024 65000 net.ipv4.tcp_max_syn_backlog=30000 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_synack_retries = 2

-/etc/my.cnf:

open_files_limit = 65536 max_connections=3600
connect_timeout=60
wait_timeout=2400
back_log=4096

-/etc/security/limits.conf

mysql soft nofile 65536 mysql hard nofile 65536 mysql soft nproc 65536 mysql hard nproc 65536

our database handle >40000 query , > 8000 connection per second 1 database machine, no sharding , connection pool. missing? 1 know may cause bottleneck? thanks.

here content in my.cnf:

[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock  skip_name_resolve = off open_files_limit = 65536  max_allowed_packet = 1073741824 expire_logs_days = 2  binlog_format = row innodb_file_per_table = 1 innodb_flush_method=o_direct  innodb_buffer_pool_instances = 16 innodb_buffer_pool_size = 43g innodb_thread_concurrency = 0  log-bin=mysql-bin server-id=1 innodb_flush_log_at_trx_commit=1 slave-skip-errors = 1062,1032  max_connections=3600 connect_timeout=60 wait_timeout=2400  back_log=4096  thread_cache_size=48  symbolic-links=0  user=mysql  [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid !includedir /etc/my.cnf.d 

so.. try increase max_connections > 8000 ?

in my.cnf

max_connections=9000