Failed Backup

cPanel, Industry leading hosting platform with world-class support.
Popetrix
Posts: 5
Joined: March 23rd, 2022, 8:34 pm

Failed Backup

Post by Popetrix »

Hi,

I am currently using JetBackup 5 with cPanel and I am having problems with a daily backup that is configured on my server. The backup takes 4 to 5 hours to finish with and error. The error is:

[ERROR] Failed transferring backup to destination "W-Repository". Error: Unable to connect to the JetIndex server.

I am using Wasabi as the destination for this backup. It has been working for month and just started failing a few weeks ago. I have contacted wasabi and they checked their logs and everything seems to be ok. This is the last message from Wasabi support:

"I have just taken a look into logs over the past couple of weeks and can see that there are no errors being returned from S3, all HTTP requests received on our end have returned 200 OK status back to the client-end.

Is Jet Backup able to provide any information on why the backup is failing? From our end, all is OK, this seems like the backup failure is due to something external to S3. Let me know if they are able to provide any info on what is resulting in this failure."

I tried to configure Dropbox as an alternative while this is fixed but I found out Dropbox is not supported on this version.

Do I need to uninstall JetBackup and install it again?

Thanks for your help and for your patience.
Popetrix
Posts: 5
Joined: March 23rd, 2022, 8:34 pm

Re: Failed Backup

Post by Popetrix »

Sorry. I forgot to mention. After this error started I cannot see any previous backups that completed successfully on any of the account on the server.

Thanks again.
JetAppsAnton
Staff Member
Staff Member
Posts: 54
Joined: March 2nd, 2019, 2:40 am

Re: Failed Backup

Post by JetAppsAnton »

Hello Popetrix,

Sorry to hear about the issue you encountered! `Error: Unable to connect to the JetIndex server.` typically occurs when the jetindexd service is not running.

You can verify the status of the jetindexd service by running the following command on your server terminal:

```
service jetindexd status
```

You may also run the following command to check if there are any errors logged for this service:

```
journalctl -u jetindexd --no-pager | tail -n 50
```

If you do observe that the jetindexd service is currently stopped, please try running the following command to restart this service as an preliminary troubleshooting step:

```
service jetindexd restart
```

If your backup job continues to fail, please don't hesitate to reach out to our 24/7 Support Team at [email protected] so that our technicians may better assist you in diagnosing your issue.

Thank You.
Anton, JetApps Team
Popetrix
Posts: 5
Joined: March 23rd, 2022, 8:34 pm

Re: Failed Backup

Post by Popetrix »

Hi,

I have found this entries using the journalctl -u jetindexd --no-pager | tail -n 50 command:

Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:688806][1171363:0
x7f00de7bb880], file:D7zkdFdcDGLiCvchMD-ub5G4pzW4SDfTkT-WUD1HVHU_snap.24_base_path.wti, txn
rollback_to_stable: __posix_open_file, 808: /usr/local/jetapps/var/lib/jetindexd/D7zkdFdcDGL
iCvchMD-ub5G4pzW4SDfTkT-WUD1HVHU_snap.24_base_path.wti: handle-open: open: Too many open fil
es
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:688903][1171363:0
x7f00de7bb880], file:D7zkdFdcDGLiCvchMD-ub5G4pzW4SDfTkT-WUD1HVHU_snap.24_base_path.wti, txn
rollback_to_stable: __rollback_to_stable_btree_apply, 1529: file:D7zkdFdcDGLiCvchMD-ub5G4pzW
4SDfTkT-WUD1HVHU_snap.24_base_path.wti: unable to open handle: Too many open files
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:689132][1171363:0
x7f00de7bb880], txn-recover: __wt_txn_recover, 1048: Recovery failed: Too many open files
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:700641][1171363:0
x7f00de7bb880], connection: __wt_cache_destroy, 364: cache server: exiting with 1308 pages i
n memory and 0 pages evicted
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:700666][1171363:0
x7f00de7bb880], connection: __wt_cache_destroy, 367: cache server: exiting with 7355931 imag
e bytes in memory
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:700669][1171363:0
x7f00de7bb880], connection: __wt_cache_destroy, 370: cache server: exiting with 7670036 byte
s in memory
Mar 24 02:38:28 anubis.generic-servers.com jetindexd[1171363]: [1648107508:700672][1171363:0
x7f00de7bb880], connection: __wt_cache_destroy, 376: cache server: exiting with 360 bytes di
rty and 1 pages dirty

Can't really understand too much but it looks that something is not right. Can you pinpoint the issue?

Thanks again for your patience.
JetAppsAnton
Staff Member
Staff Member
Posts: 54
Joined: March 2nd, 2019, 2:40 am

Re: Failed Backup

Post by JetAppsAnton »

Hello Popetrix,

Thank you for providing us with a snippet of the jetindexd service logs.

For our next step, could you please try updating your JetBackup installation to our current EDGE version (v5.2.7) as we introduced improvements in the stability and reliability of our indexing engine (JetIndex). To do so, please do the following:

Code: Select all

# Change the release tier to edge(if desired)
jetapps --chtier jetbackup5-{__PANEL_TYPE__} edge

# Update JetBackup manually:
jetapps -u jetbackup5-{__PANEL_TYPE__}

# Change tier back to STABLE (for future stable tier updates)
jetapps --chtier jetbackup5-{__PANEL_TYPE__} stable
please change the {__PANEL_TYPE__} to your appropriate panel (cpanel, directadmin, linux)

If you continue to observe this issue, please try the following TEMPORARY workaround next:

As a workaround, the `jetindexd process limit` can be increased. This can be done by adding the following lines to the jetindexd file `/usr/lib/systemd/system/jetindexd.service` under the "service" section.

Code: Select all

LimitNOFILE=65535
LimitNPROC=65535
Once the change is made, the service should be restarted to reflect the changes. You can restart the service using the following commands,

Code: Select all

# systemctl daemon-reload
# systemctl restart jetindexd
We look forward to hearing your update, and just to reiterate, please don't hesitate to reach out to our 24/7 Support Team at [email protected] so that our technicians may better assist you in diagnosing your issue if you continue to experience issues.
Anton, JetApps Team
Popetrix
Posts: 5
Joined: March 23rd, 2022, 8:34 pm

Re: Failed Backup

Post by Popetrix »

Hi,

When I ran the command "jetapps -u jetbackup5-cpanel", it said that the version was up-to-date already.

After adding:

LimitNOFILE=65535
LimitNPROC=65535

and,

# systemctl daemon-reload
# systemctl restart jetindexd

I ran a backup job and it finished with the failed status.

Thanks.
JetAppsAnton
Staff Member
Staff Member
Posts: 54
Joined: March 2nd, 2019, 2:40 am

Re: Failed Backup

Post by JetAppsAnton »

Hello Popetrix,

Thank you for following up and providing us with an update and sorry to hear that you are still having issues running your backup job. Please kindly reach out to our 24/7 Support Team at [email protected] so that our technicians may further assist in diagnosing your issue.

Best Regards,
Anton, JetApps Team
Popetrix
Posts: 5
Joined: March 23rd, 2022, 8:34 pm

Re: Failed Backup

Post by Popetrix »

Thanks!

Done. I posted this conversation in the email.
JetAppsAnton
Staff Member
Staff Member
Posts: 54
Joined: March 2nd, 2019, 2:40 am

Re: Failed Backup

Post by JetAppsAnton »

Hello Popetrix,

Thank you for opening up a ticket with us and we will update this forum thread as well with our support team's findings and corresponding fix.

Best Regards
Anton, JetApps Team
JetAppsAnton
Staff Member
Staff Member
Posts: 54
Joined: March 2nd, 2019, 2:40 am

Re: Failed Backup

Post by JetAppsAnton »

Hello,

We'd just like to follow up with the information we were able to gather so far and it appears that the issue was caused by an older version of our jetindexd service (v1.1.0-1) which needed to be updated to v1.1.1-1

Thank You.
Anton, JetApps Team
Post Reply