API paginater problem

API/SDK/Tea/HelmChart/Terraform
1
  • Gitea version (or commit ref): 1.9.4
  • Git version: 2.17.1
  • Operating system: ubuntu 18.04
  • Database:
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant

Description

Hello everyone. Glad to be here.
I did not have experience in go lang. I had a workaround on the codes but could not fix this,

Summary

During the import of Gitea repositories into GitLab, some issues may be skipped over due to query string parameters changing during API pagination requests. Specialty in the last pages.

Steps to reproduce

You can reface this when importing repos to the GitLab. On some repositories with many numbers of issues and PRs, the pagination API responses next and prev page numbers incorrect

Logs

Here i’\ insert the logs which local Gitea server shows when importing repositories from Gitea to the GitLab:

10.242.12.2 - - [26/Nov/2019:07:45:18 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=2&per_page=100&sort=created&state=all HTTP/1.1" 200 9886 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:19 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=3&per_page=100&sort=created&state=all HTTP/1.1" 200 13483 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:20 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=4&per_page=100&sort=created&state=all HTTP/1.1" 200 9115 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:21 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=5&per_page=100&sort=created&state=all HTTP/1.1" 200 8913 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:23 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all HTTP/1.1" 200 8991 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:24 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all HTTP/1.1" 200 8268 "-" "Octokit Ruby Gem 4.9.0"

The problem is that more pages still available but Gitea responses the last page is page 7. To show more details, let me request a currect scenario on page 6 something like:

request

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all

response header

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all&state=all>; rel="next",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all&state=all>; rel="last",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=5&per_page=100&sort=created&state=all&state=all>; rel="prev"

But the last page is not page 7, let me request an incorrect scenario on page 7 and 8 something like:

request page 7

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all

response header page 7

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all&state=all>; rel="prev"

request page 8

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=8&per_page=100&sort=created&state=all

response header page 8

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all&state=all>; rel="prev"

but we can see still page 8 is valid and has data, the response body of page 8:

[
    {
        "id": 6043,
        "url": "https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues/15",
        "number": 15,
        "user": {
            "id": 1,
            "login": "admin",
            "full_name": "admin",
            "email": "***@***.com",
            "avatar_url": "https://secure.gravatar.com/avatar/***",
            "language": "en-US",
            "is_admin": true,
            "last_login": "2019-11-25T06:40:56Z",
            "created": "2019-08-01T13:23:17Z",
            "username": "admin"
        },
        "title": "Set limits",
 ....

PS:

  • I know some query strings like per_page, direction, etc. are not handled by Gittea, but this does not result in a problem. The requests made by GitLab.
  • I know the problem is result of reponsed header link field which set the next,prev, first and last pages.
2

Nobody can help me?
We are in problem in our organazation

3

I’m sorry @debMan, this sounds like a bug. There’s work in progress to improve GitLab import in 1.11.0, but it’s not in the main branch yet. Nevertheless, you should file an issue in Issues · go-gitea/gitea · GitHub and maybe reference your issue in that PR.