Orbiter History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 651.966913ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:10:25 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 651.982122ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 660.490248ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:09:36 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 660.516039ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.671278333s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.671301496s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 654.07489ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:08:39 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 654.099385ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.686841384s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:08:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.686860902s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 652.245557ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:08:08 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 652.279352ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.909425369s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:08:07 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.909466127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.684160653s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:07:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.684188047s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 661.425194ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:07:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 661.45444ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 652.156639ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:07:30 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 652.180894ms core satellite us1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 430.345455ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:07:22 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 430.374771ms object-mount
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.753163303s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.753187158s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 659.077944ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:07:09 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 659.103784ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.92298396s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:06:58 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.923008257s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 664.350696ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:06:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 664.37345ms core satellite us1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 430.418536ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:06:31 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 430.44087ms object-mount
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 653.067716ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:06:21 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 653.091471ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.676368744s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:06:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.676395167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.879729463s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:05:57 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.879750753s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 652.365456ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:05:52 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 652.387228ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.352487983s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:05:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.352549089s core uplink us-select-1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 432.655345ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:05:18 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 432.681396ms object-mount
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 665.639246ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:05:04 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 665.663723ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.328636327s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:04:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.328667217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.891566641s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:04:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.891582551s core uplink us-select-1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 423.22537ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:04:53 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 423.252432ms object-mount
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 666.569179ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:04:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 666.599408ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.656864727s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:04:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.656879665s core uplink us-select-1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 421.376262ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:04:18 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 421.397001ms object-mount
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 1.886925334s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:04:16 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.886952556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.138034652s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.138055531s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 650.913245ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:03:48 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 650.949233ms core satellite us1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 425.272456ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:03:34 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 425.293296ms object-mount
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 1.896038656s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:03:30 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.896060808s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 652.785739ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:03:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 652.803742ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.655643525s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.655666489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 1.88721609s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:03:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.887244365s core uplink us-select-1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 422.631357ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:02:38 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 422.644282ms object-mount
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 657.008127ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:02:12 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 657.036672ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.66803939s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.668067224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 1.892090672s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:02:03 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.892119086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.880043212s trace: https://jaeger.dev.storj.io/trace/3821857227b01f0d FAIL (2026-05-13 09:02:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.880079231s core uplink us1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 431.645938ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:01:38 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 431.669873ms object-mount
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 1.898051407s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:01:24 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.898075432s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 661.566764ms remote: "tcp:34.4.32.59:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/68a242383b75c35d FAIL (2026-05-13 09:01:12 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 661.595229ms core satellite us1
checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 432.031953ms target: http://timesync.objectmount.io/health trace: https://jaeger.dev.storj.io/trace/498a8e7089f2c4a8 FAIL (2026-05-13 09:01:12 UTC) orbiter warning: Get "http://timesync.objectmount.io/health": dial tcp 35.209.219.1:80: i/o timeout Duration: 432.068773ms object-mount
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.964861916s trace: https://jaeger.dev.storj.io/trace/3821857227b01f0d FAIL (2026-05-13 09:01:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.96491639s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 1.913211855s trace: https://jaeger.dev.storj.io/trace/6b68594ae31476df FAIL (2026-05-13 09:00:57 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.913237683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.758182457s trace: https://jaeger.dev.storj.io/trace/f253484fd40b3cf FAIL (2026-05-13 09:00:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 52.886159919s) [upload=288931]: Starting upload May 13 08:23:42.365411428 (301h35m52.886235173s) [upload=288931]: Done starting upload May 13 08:23:42.365500037 (301h35m52.886323782s) [upload=288931]: Uploading segments... May 13 08:23:42.365647579 (301h35m52.886471334s) [upload=288931]: Got next segment. Inline: false May 13 08:23:42.365683176 (301h35m52.886506911s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 08:23:42.365687885 (301h35m52.88651163s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 08:23:42.598383896 (301h35m53.119207641s) [seg_pos={0 0}, upload=288931]: Begin upload segment... May 13 08:23:42.598464609 (301h35m53.119288344s) [seg_pos={0 0}, upload=288931]: Done begin upload segment. May 13 08:23:42.598489827 (301h35m53.119313562s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.598584227 (301h35m53.119407972s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599099290 (301h35m53.119923035s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599448427 (301h35m53.120272172s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.599785570 (301h35m53.120609315s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600089149 (301h35m53.120912894s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600393219 (301h35m53.121216964s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.600679686 (301h35m53.121503431s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601010748 (301h35m53.121834493s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.601837916 (301h35m53.122661651s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.602900945 (301h35m53.12372469s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.603751337 (301h35m53.124575072s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.604687004 (301h35m53.125510749s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.605665119 (301h35m53.126488864s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.606795627 (301h35m53.127619372s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:42.607929281 (301h35m53.128753026s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Uploading piece... May 13 08:23:44.318577649 (301h35m54.839401394s) [upload=288931]: Next returned nil segment May 13 08:23:44.318598147 (301h35m54.839421892s) [upload=288931]: Waiting for error group managing segments... May 13 08:23:44.604157373 (301h35m55.124981108s) [noise=true , address=145.79.227.50:4374 , piece_id=VU7FGWLJIQ6BJWLWR4V3FZ2OOKWZ4XUXXIK6UTIOXG4JQRS5QAVQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.616951337 (301h35m55.137775082s) [noise=true , address=145.79.227.45:4233 , piece_id=25YQOKEQLNYGD2DZBPWQD5HMUKBXEA2YMYVLRZWQVHCPXWKFOA3Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.626138077 (301h35m55.146961822s) [noise=true , address=145.79.227.7:4297 , piece_id=W2ZWVO5KC5ZJMC5LZKQ3ZBGEOBGE2Z7DN6BLRZCGPJ2LWDFEIAGA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627298322 (301h35m55.148122067s) [noise=true , address=145.79.227.3:4152 , piece_id=BZGI2SX3KLTQ5L74GQTXWJ3MXN5KLLFZKMPES5FRGGCLUD7XXV3A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627592093 (301h35m55.148415838s) [noise=true , address=145.79.227.52:4447 , piece_id=XTJJIUVCL2WH5TJ4CZRJAUETVQIYYB55KALFX46YG2FPOMLLOPUA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.627893708 (301h35m55.148717453s) [noise=true , address=145.79.227.6:4263 , piece_id=MPF6TBBYJ3IQVVBMS36BIV6PAITZ25AYM3TJAI2KO7XBZB2FN7WQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.630105298 (301h35m55.150929043s) [noise=true , address=145.79.227.39:4017 , piece_id=H2XFKXRSXEAG4WM7XDSYFLSHUOECVFXR47PBYVHW4DZLKJXWLU5A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.635153522 (301h35m55.155977267s) [noise=true , address=145.79.227.25:4619 , piece_id=3EFGZMPN4XP4F4U5OKK5VIYD44FT3WXQB73NZQL4LFYVQMTSUIVA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.639359247 (301h35m55.160182992s) [noise=true , address=145.79.227.4:4185 , piece_id=73P5QCJDK5IESMP53DQNMZHRSOHFDVYXEM6AWW7CWR7PDDKH4EXQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.642768653 (301h35m55.163592388s) [noise=true , address=145.79.227.31:4798 , piece_id=2ZEX5FUWBC2NYJJCGZ5UMPBGKKXNGXFH27CMWB7I55BWAQ4X33XQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.669518538 (301h35m55.190342283s) [noise=true , address=145.79.227.13:4804 , piece_id=GAB5UUUHLI3P6YSFVQ6HHBTPCTLQOPE6THKSHKZPSAHLI7G2QFHA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.673346913 (301h35m55.194170658s) [noise=true , address=145.79.227.26:4654 , piece_id=OYHDYSBP5LMUP6ZQONZNELXVOY53ZSJMG33ONRQLLFGYPFKR3P5Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722674662 (301h35m55.243498407s) [noise=true , address=77.90.31.15:4701 , piece_id=XQI7GRPTXPE35WKWKEQX72RSEQ74NMF7645CXJA2E5FKWWKHEI2A, seg_pos={0 0}, upload=288931]: Done uploading piece. err: <nil> May 13 08:23:44.722697015 (301h35m55.24352076s) [seg_pos={0 0}, upload=288931]: Satisfied cohort requirements after 13 pieces May 13 08:23:44.722987831 (301h35m55.243811566s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.722997429 (301h35m55.243821174s) [noise=true , address=77.90.31.13:4626 , piece_id=YS5MLLTJQBYQ6VCEKMYUNBVEIKVQAC7L22PRY2BRMLPA25IR3D7Q, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723038947 (301h35m55.243862692s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kwa9zxc9LDp78psKPNV55aP5RYvtcQi1NjphTkXBWRugwhg7s): protocol: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->45.91.116.4:4972: use of closed network connection May 13 08:23:44.723054898 (301h35m55.243878643s) [noise=true , address=45.91.116.4:4972 , piece_id=GHKQU3ZRVRBRIQ3SYS42DGTC2K3VQ3UHAGLBKST7QZYAH5LSEXLA, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723109372 (301h35m55.243933117s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF May 13 08:23:44.723116526 (301h35m55.243940281s) [noise=true , address=145.79.227.49:4340 , piece_id=VH6N2FJ76D6UQS3DBHRDBXVMMMHVL73MS5MMLF4X6ITFGIRZYMZQ, seg_pos={0 0}, upload=288931]: Long tail cancellation due to optimal threshold May 13 08:23:44.723129540 (301h35m55.243953285s) [seg_pos={0 0}, upload=288931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 08:23:44.723189946 (301h35m55.244013691s) [upload=288931]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 08:23:44.723193784 (301h35m55.244017519s) [upload=288931]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 08:24:49.413522848 (301h36m59.934346603s) [upload=288931]: Done uploading segments... May 13 08:24:49.413641645 (301h36m59.93446539s) [upload=288931]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.758211462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.909436333s trace: https://jaeger.dev.storj.io/trace/3821857227b01f0d FAIL (2026-05-13 09:00:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.909452273s core uplink us1

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "http://timesync.objectmount.io/health", ExpectedStatus: 200} failure (antiblip window 3/60): (http.timesync_objectmount_io_health)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)