Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X...........X...X....X................X.X......XXXXX.XXXX] duration: 30.000991835s trace: https://jaeger.dev.storj.io/trace/1a08fa549095e08e FAIL (2026-01-12 09:57:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001002215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.731429614s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.731460514s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.000558589s trace: https://jaeger.dev.storj.io/trace/5558dbf415d2452 FAIL (2026-01-12 09:57:06 UTC) orbiter warning: context deadline exceeded Duration: 5.000582294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X...........X...X....X................X.X......XXXXX.XXX] duration: 30.001202821s trace: https://jaeger.dev.storj.io/trace/20092f9904e59b6 FAIL (2026-01-12 09:56:52 UTC) orbiter warning: context deadline exceeded Duration: 30.001226495s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.002221816s trace: https://jaeger.dev.storj.io/trace/483739c5f34a8f8 FAIL (2026-01-12 09:56:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002244798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 5.854332576s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.854375015s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X....................X................] duration: 18.957886286s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:56:14 UTC) orbiter warning: context deadline exceeded Duration: 18.957922455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001861699s trace: https://jaeger.dev.storj.io/trace/7cb382c080dd620 FAIL (2026-01-12 09:56:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001886577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X...........X...X....X................X.X......XXXXX.XX] duration: 30.000948754s trace: https://jaeger.dev.storj.io/trace/30feed5c39e03af6 FAIL (2026-01-12 09:55:49 UTC) orbiter warning: context deadline exceeded Duration: 30.000974543s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.001012942s trace: https://jaeger.dev.storj.io/trace/42950fbb9bd7d37 FAIL (2026-01-12 09:55:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001040135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....................X...............] duration: 20.300216942s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:55:34 UTC) orbiter warning: context deadline exceeded Duration: 20.300243782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.740172395s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:55:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.740197514s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.001712968s trace: https://jaeger.dev.storj.io/trace/bc0b8da120bb90 FAIL (2026-01-12 09:55:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001737254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.851648779s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:55:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.85167529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X....................X..............] duration: 24.271237099s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:54:51 UTC) orbiter warning: context deadline exceeded Duration: 24.271267498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X...........X...X....X................X.X......XXXXX.X] duration: 30.001571205s trace: https://jaeger.dev.storj.io/trace/6faaf30e94de39ed FAIL (2026-01-12 09:54:45 UTC) orbiter warning: context deadline exceeded Duration: 30.00159457s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.00179304s trace: https://jaeger.dev.storj.io/trace/9e2b386c2afb301 FAIL (2026-01-12 09:54:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001816015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....................X.............] duration: 19.143793659s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:54:27 UTC) orbiter warning: context deadline exceeded Duration: 19.143807476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.71805884s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.718088496s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....................X............] duration: 28.926827036s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:54:07 UTC) orbiter warning: context deadline exceeded Duration: 28.926853066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 5.738630217s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.738660184s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.001492932s trace: https://jaeger.dev.storj.io/trace/44a7fe80ff4b1d1a FAIL (2026-01-12 09:53:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001516376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X...........X...X....X................X.X......XXXXX.] duration: 19.243178013s trace: https://jaeger.dev.storj.io/trace/9c43aa94bacceec FAIL (2026-01-12 09:53:37 UTC) orbiter warning: context deadline exceeded Duration: 19.243191308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 5.798480415s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:53:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.798500844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000659886s trace: https://jaeger.dev.storj.io/trace/3aed74d1c78b178a FAIL (2026-01-12 09:53:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000682347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X...........X...X....X................X.X......XXXXX] duration: 30.001015403s trace: https://jaeger.dev.storj.io/trace/9c43aa94bacceec FAIL (2026-01-12 09:53:18 UTC) orbiter warning: context deadline exceeded Duration: 30.001034278s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....................X...........] duration: 23.18380637s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:52:50 UTC) orbiter warning: context deadline exceeded Duration: 23.183826749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X...........X...X....X................X.X......XXXX] duration: 30.001230602s trace: https://jaeger.dev.storj.io/trace/507b8152bac1832a FAIL (2026-01-12 09:52:40 UTC) orbiter warning: context deadline exceeded Duration: 30.001240681s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001474721s trace: https://jaeger.dev.storj.io/trace/2b79cb48242d9e9 FAIL (2026-01-12 09:52:35 UTC) orbiter warning: context deadline exceeded Duration: 5.001496793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.707231109s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:52:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.707259613s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X...........X...X....X................X.X......XXX] duration: 30.001326563s trace: https://jaeger.dev.storj.io/trace/5074e637b6a6b470 FAIL (2026-01-12 09:52:09 UTC) orbiter warning: context deadline exceeded Duration: 30.001339027s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X....................X..........] duration: 18.48942617s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:52:00 UTC) orbiter warning: context deadline exceeded Duration: 18.489441117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.001137493s trace: https://jaeger.dev.storj.io/trace/775f79da8a4cac60 FAIL (2026-01-12 09:51:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001164684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.75924483s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:51:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.75927195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....................X.........] duration: 40.013748426s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:51:42 UTC) orbiter warning: context deadline exceeded Duration: 40.013777261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X...........X...X....X................X.X......XX] duration: 30.002046557s trace: https://jaeger.dev.storj.io/trace/7bb790f9e145c21c FAIL (2026-01-12 09:51:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002071855s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.000892907s trace: https://jaeger.dev.storj.io/trace/36b28e9828036567 FAIL (2026-01-12 09:51:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000916052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.739738129s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:51:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.739764418s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X...........X...X....X................X.X......X] duration: 30.001195586s trace: https://jaeger.dev.storj.io/trace/2009f4e110a07afd FAIL (2026-01-12 09:50:46 UTC) orbiter warning: context deadline exceeded Duration: 30.001221574s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.000889032s trace: https://jaeger.dev.storj.io/trace/23d4587f60dd6fa4 FAIL (2026-01-12 09:50:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000916575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....................X........] duration: 20.897049263s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:50:20 UTC) orbiter warning: context deadline exceeded Duration: 20.897077015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X...........X...X....X................X.X......] duration: 27.104169758s trace: https://jaeger.dev.storj.io/trace/4a1812e6fb4db0a8 FAIL (2026-01-12 09:50:14 UTC) orbiter warning: context deadline exceeded Duration: 27.10420273s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 6.030543196s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 6.030580327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....................X.......] duration: 21.748949507s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:49:48 UTC) orbiter warning: context deadline exceeded Duration: 21.748974777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X...........X...X....X................X.X.....] duration: 28.095837705s trace: https://jaeger.dev.storj.io/trace/4a1812e6fb4db0a8 FAIL (2026-01-12 09:49:32 UTC) orbiter warning: context deadline exceeded Duration: 28.095866439s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 5.001012227s trace: https://jaeger.dev.storj.io/trace/5be2211498def338 FAIL (2026-01-12 09:49:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001034109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.779250878s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:49:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.779276076s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X....................X......] duration: 26.23868652s trace: https://jaeger.dev.storj.io/trace/5517fdce0b8717bf FAIL (2026-01-12 09:48:48 UTC) orbiter warning: context deadline exceeded Duration: 26.23871303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 5.720691487s trace: https://jaeger.dev.storj.io/trace/ab64659e76d444f FAIL (2026-01-12 09:48:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA) context deadline exceeded; piecestore: (Node ID: 19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, Piece ID: MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA) context deadline exceeded ======LOGS====== Jan 12 09:16:56.792600762 (905h47m56.663683807s) [upload=792826]: Starting upload Jan 12 09:16:56.792703167 (905h47m56.663786212s) [upload=792826]: Done starting upload Jan 12 09:16:56.792791775 (905h47m56.66387485s) [upload=792826]: Uploading segments... Jan 12 09:16:56.792986346 (905h47m56.664069391s) [upload=792826]: Got next segment. Inline: false Jan 12 09:16:56.793042272 (905h47m56.664125317s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 09:16:56.793048945 (905h47m56.66413198s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 09:16:57.303000354 (905h47m57.174083399s) [seg_pos={0 0}, upload=792826]: Begin upload segment... Jan 12 09:16:57.303050018 (905h47m57.174133063s) [seg_pos={0 0}, upload=792826]: Done begin upload segment. Jan 12 09:16:57.303099402 (905h47m57.174182438s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303125853 (905h47m57.174208888s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.303863027 (905h47m57.174946062s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.304692776 (905h47m57.175775821s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305125821 (905h47m57.176208856s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.305559506 (905h47m57.176642541s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.306039110 (905h47m57.177122145s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307511213 (905h47m57.178594248s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.307674754 (905h47m57.178757789s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309207323 (905h47m57.180290368s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.309324225 (905h47m57.18040727s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.310985969 (905h47m57.182069004s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.311179898 (905h47m57.182262943s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.313122637 (905h47m57.184205683s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.315131393 (905h47m57.186214438s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:57.319394119 (905h47m57.190477154s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Uploading piece... Jan 12 09:16:59.087297100 (905h47m58.958380135s) [upload=792826]: Next returned nil segment Jan 12 09:16:59.087310024 (905h47m58.958393059s) [upload=792826]: Waiting for error group managing segments... Jan 12 09:16:59.429705807 (905h47m59.300788842s) [noise=true , address=77.90.31.11:4553 , piece_id=YUHJ6FJACICRGDJ6RPZQVADCT6UTLCLDHIJZZ7MTGJOXAJLEAQHA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.448043334 (905h47m59.319126379s) [noise=true , address=77.90.31.14:4663 , piece_id=Q7QJLRLXVHJOBIUWVX6A7APFL2UE2QKXRQQCT767GGS5QDR4P75Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.456028489 (905h47m59.327111524s) [noise=true , address=145.79.227.5:4223 , piece_id=NT4FP5F3Z7VI5BLGRSLLRVCGRCEE2XFNZONKECRPOGITVAW5HYOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.486510770 (905h47m59.357593806s) [noise=true , address=145.79.227.43:4157 , piece_id=2QI5I6K5V6ZWQYFCCKNJOMDFTFXH65TC44BWPLYXWAWQHJQHHJ7A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.494721013 (905h47m59.365804058s) [noise=true , address=145.79.227.44:4193 , piece_id=WLUJ7242QP2MJ5REPRCYK3LIKIWLBUH2JVUX6QXGIAPW4IKFANAQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.542756890 (905h47m59.413839965s) [noise=true , address=145.79.227.15:4873 , piece_id=DGDGA54UVGUHV2P6ILI5FTDQX2VG5XMERS74XJADER3VCXV6OGEA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.555273382 (905h47m59.426356417s) [noise=true , address=145.79.227.46:4601 , piece_id=ILQV7LYBEVQL2IG3SDDH2BUJ5JIXNDEMPT2JLYJNVLHT64IBZEKQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.619530646 (905h47m59.490613681s) [noise=true , address=145.79.227.45:4231 , piece_id=OJ3KP5XXUTXSI5IG7BMVIF46E5QXVLCVFYYLPWYANYZNLWG5NGOA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.797964675 (905h47m59.669047721s) [noise=true , address=145.79.227.31:4799 , piece_id=ECBF2XWHBWEOERAIOAQAI6RUY32DDQLLKLDPKWZX4E537K7X7K5Q, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.808633430 (905h47m59.679716475s) [noise=true , address=145.79.227.51:4409 , piece_id=AX7DTBGHU2HAKOZ4FAH3XKYQPMK4L3N7MTQFREPIMVGGROSUF2FQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:16:59.943655669 (905h47m59.814738714s) [noise=true , address=145.79.227.23:4547 , piece_id=LVXZYBX2D6JDRQQN6JPVZHNQIVP4G5BW3TCETP23W22MHCEZOQKA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175177895 (905h48m0.046260931s) [noise=true , address=145.79.227.18:4007 , piece_id=MTRREB3KN6EXE3SAIFZ5HBHPQB7LEQQSRUVI7ANOBCTMUEH7X36A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175884611 (905h48m0.046967656s) [noise=true , address=145.79.227.33:4871 , piece_id=CUYB4D5YJ7KNDB4DC3YAF53PTYXQO2UWZPFUYQLKPYHZZJZ5D4KQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: <nil> Jan 12 09:17:00.175909178 (905h48m0.046992223s) [seg_pos={0 0}, upload=792826]: Segment reached optimal threshold of 13 pieces. Jan 12 09:17:00.176276938 (905h48m0.047359973s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55404->145.79.227.17:4970: use of closed network connection Jan 12 09:17:00.176286415 (905h48m0.04736945s) [noise=true , address=145.79.227.17:4970 , piece_id=B23BICB6N7NQJJ3JA7KCMDOUVIB36XFQRUIBFEZN3Z7BHP2LAL2A, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176336792 (905h48m0.047419847s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176347923 (905h48m0.047430968s) [noise=true , address=145.79.227.21:4474 , piece_id=5WNFIMZC2IBSLSYYB3A3KYEIRW32CZGIV2SVR7TNNQMJAKULPLUA, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176508509 (905h48m0.047591554s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Jan 12 09:17:00.176519770 (905h48m0.047602805s) [noise=true , address=145.79.227.8:4330 , piece_id=I6DKLETV33CG3E77SYDQVUPNI5FYG3NQNBIHWV3TCHH6BLBDGOSQ, seg_pos={0 0}, upload=792826]: Long tail cancellation due to optimal threshold Jan 12 09:17:00.176538286 (905h48m0.047621321s) [seg_pos={0 0}, upload=792826]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 09:17:00.176698160 (905h48m0.047781195s) [upload=792826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 09:17:00.176704502 (905h48m0.047787537s) [upload=792826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 09:17:00.493070711 (905h48m0.364153756s) [upload=792826]: Done uploading segments... Jan 12 09:17:00.493089967 (905h48m0.364173002s) [upload=792826]: Upload finished. err: <nil> Duration: 5.720717187s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 5.001845428s trace: https://jaeger.dev.storj.io/trace/69d34d761e6c2332 FAIL (2026-01-12 09:48:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001860909s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)