Gitea stopped running (due to database error "Got an error reading communication packets")

Install/Maintain/Configure
1

Hello,

Gitea worked for some weeks, but suddenly stopped working yesterday (or maybe some days before). I’ve running it within docker

version: “3”

networks:
gitea:
external: false

services:
server:
image: gitea/gitea:latest
environment:
- USER_UID=1000
- USER_GID=1000
restart: always
networks:
- gitea
volumes:
- ./data:/data
- /srv/services/git:/persistent
- tmpdata:/tmpdata
ports:
- “8083:8083”
- “2222:2222”
depends_on:
- database

database:
image: mariadb
restart: always
environment:

 MYSQL_DATABASE: gitea
 MYSQL_USER: ${MYSQL_USER}
 MYSQL_PASSWORD: ${MYSQL_PASSWORD}
 MYSQL_RANDOM_ROOT_PASSWORD: 'yes'

networks:
- gitea
volumes:
- mariadb:/var/lib/mysql
volumes:
mariadb:
tmpdata:

I now get these database logs:

2018-10-06 5:37:50 0 [Note] mysqld (mysqld 10.3.10-MariaDB-1:10.3.10+maria~bionic) starting as process 1 …
2018-10-06 5:37:50 0 [Note] InnoDB: Using Linux native AIO
2018-10-06 5:37:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-10-06 5:37:50 0 [Note] InnoDB: Uses event mutexes
2018-10-06 5:37:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2018-10-06 5:37:50 0 [Note] InnoDB: Number of pools: 1
2018-10-06 5:37:50 0 [Note] InnoDB: Using generic crc32 instructions
2018-10-06 5:37:50 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2018-10-06 5:37:50 0 [Note] InnoDB: Completed initialization of buffer pool
2018-10-06 5:37:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-10-06 5:37:51 0 [Note] InnoDB: Upgrading redo log: 2*50331648 bytes; LSN=8019001
2018-10-06 5:37:51 0 [Note] InnoDB: Starting to delete and rewrite log files.
2018-10-06 5:37:51 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes
2018-10-06 5:37:51 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes
2018-10-06 5:37:51 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2018-10-06 5:37:51 0 [Note] InnoDB: New log files created, LSN=8019001
2018-10-06 5:37:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-10-06 5:37:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-10-06 5:37:51 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2018-10-06 5:37:51 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2018-10-06 5:37:51 0 [Note] InnoDB: 10.3.10 started; log sequence number 8019001; transaction id 39852
2018-10-06 5:37:51 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-10-06 5:37:51 0 [Note] Plugin ‘FEEDBACK’ is disabled.
2018-10-06 5:37:51 0 [Note] Server socket created on IP: ‘::’.
2018-10-06 5:37:51 0 [Warning] ‘proxies_priv’ entry ‘@% root@595badc4182f’ ignored in --skip-name-resolve mode.
2018-10-06 5:37:51 0 [Note] Reading of all Master_info entries succeded
2018-10-06 5:37:51 0 [Note] Added new Master_info ‘’ to hash table
2018-10-06 5:37:51 0 [Note] mysqld: ready for connections.
Version: ‘10.3.10-MariaDB-1:10.3.10+maria~bionic’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
2018-10-06 5:37:51 0 [Note] InnoDB: Buffer pool(s) load completed at 181006 5:37:51
2018-10-06 5:37:55 10 [Warning] Aborted connection 10 to db: ‘name’ user: ‘name’ host: ‘ipaddress’ (Got an error reading communication packets)
2018-10-06 5:37:55 9 [Warning] Aborted connection 9 to db: ‘name’ user: ‘name’ host: ‘ipaddress’ (Got an error reading communication packets)
2018-10-06 5:37:59 13 [Warning] Aborted connection 13 to db: ‘name’ user: ‘name’ host: ‘ipaddress’ (Got an error reading communication packets)
2018-10-06 5:37:59 12 [Warning] Aborted connection 12 to db: ‘name’ user: ‘name’ host: ‘ipaddress’ (Got an error reading communication packets)

Last lines repeat endlessly and I can’t connect to the webinterface.
If I connect to the docker image with exec and try to do selects within tables everything works fine.

The only thing which comes into my mind, what I’ve done the last days is following:

  • I did a fresh install on my host system 2 weeks ago and 2-3 days ago I noticed that utc-time was still set, so I changed it to my local time.
  • I did also a reboot, to see if this does change something.

Any help is appreciated,
Thanks,
Cyber1000

1 Like
2

Problem solved:

  • I moved /data/gitea/ssh/gogs.rsa some time ago on my host and since I did not reboot until now, I didn’t notice the problem.
  • within docker, there were problems creating a new /data/gitea/ssh/gogs.rsa (actually ssh-dir was not writable within docker)
  • Though it’s somehow funny that massive loads of connections are opened to database (and it seems that they are not closed properly), bringing my small fileserver dual-core atom to more than 120% CPU-usage (a whole and a part of another core) …
3

Thanks for sharing your Dockerfile. Question. How are you managing your DB password? Is that kept in the Volume you’re using our somewhere else?

4

The file I’ve shared is a docker-compose.yml and I have a .env file in the same folder, which consists of simple key-value pairs (Use an environment file in Docker Compose | Docker Docs)

MYSQL_USER=…
MYSQL_PASSWORD=…

docker-compose.yml (and possible small bash scripts) of all my docker-containers are backuped/versioned within git/gitea on my fileserver, since they are themselve passwordless I can save them anywhere, but .env is never stored anywhere else than in the same directory as docker-compose.yml and only accessible by root. In case of harddiskfailure I’ll have to recreate them manually.

1 Like
5

That’s a solid approach. Thanks for sharing. If you care to write it down I’m happy to provide you a static wcms solution which scales with your traffic.

6

I have got the same problem since tuesday, but i did not touch /data/gitea/ssh/gogs.rsa. Then I found out that it seems the latest docker build is broken! Only back to 1.6.1 and restore from backup solved my problem!

I’d say, there is a new bug in gitea since about 1.6.2 or 1.6.3!

My configuration in docker swarm:

version: '3.3'

services:

  mysql:
    image: mysql:5.7
    volumes:
      - type: bind
        source: /srv/volumes/gitea-mrw-sh/mysql
        target: /var/lib/mysql
    environment:
      - 'MYSQL_ROOT_PASSWORD=****'
      - 'MYSQL_USER=gitea'
      - 'MYSQL_DATABASE=gitea'
      - 'MYSQL_PASSWORD=****'
    command: --log-bin=/var/lib/mysql/mysql-bin.log --binlog-format=ROW --server-id=1
    deploy:
      resources:
        reservations:
          cpus: '0.01'
          memory: 5M
        limits:
          memory: 1G

  smtp:
    image: mwaeckerlin/smtp-relay-tls
    environment:
      - MAILHOST=mrw.sh
    deploy:
      resources:
        reservations:
          cpus: '0.01'
          memory: 5M
        limits:
          memory: 500M

  gitea:
    image: gitea/gitea:1.6.0
    ports:
      - 8012:3000
    labels:
      - 'url=https://mrw.sh'
    volumes:
      - type: bind
        source: /srv/volumes/gitea-mrw-sh/gitea
        target: /data
    depends_on:
      - mysql
    deploy:
      resources:
        reservations:
          cpus: '0.01'
          memory: 5M
        limits:
          memory: 1G
7

maybe this issue refer to go-sql-driver/ mysql


anyone to check and fix it?