Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...X.XXX] duration: 5.001821556s trace: https://jaeger.dev.storj.io/trace/441714a9baabdec FAIL (2026-01-09 09:40:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001848368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.845857981s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.845887356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...X.XX] duration: 5.001163974s trace: https://jaeger.dev.storj.io/trace/739bac609bf3739 FAIL (2026-01-09 09:40:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001192808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...........XXXX...............X.X.....X....................] duration: 19.073413336s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:39:53 UTC) orbiter warning: context deadline exceeded Duration: 19.073454213s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 18.043206242s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:39:47 UTC) orbiter warning: context deadline exceeded Duration: 18.043230529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.763360887s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.763391366s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...X.X] duration: 5.001233438s trace: https://jaeger.dev.storj.io/trace/2bda0900c17109d FAIL (2026-01-09 09:39:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001259488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 19.465139817s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:39:18 UTC) orbiter warning: context deadline exceeded Duration: 19.465154374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.882592977s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:39:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.882626902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.785291584s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:38:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.78532111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 19.495603143s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:38:47 UTC) orbiter warning: context deadline exceeded Duration: 19.495626838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...........XXXX...............X.X.....X...................] duration: 11.030551619s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:38:38 UTC) orbiter warning: context deadline exceeded Duration: 11.030601055s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...X.] duration: 3.878158088s trace: https://jaeger.dev.storj.io/trace/60740405c6141389 FAIL (2026-01-09 09:38:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.878187272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.824427509s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.824451786s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...X] duration: 5.001251582s trace: https://jaeger.dev.storj.io/trace/60740405c6141389 FAIL (2026-01-09 09:38:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00130296s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 17.181040164s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:38:04 UTC) orbiter warning: context deadline exceeded Duration: 17.18106851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X...........XXXX...............X.X.....X..................] duration: 7.238497407s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:38:00 UTC) orbiter warning: context deadline exceeded Duration: 7.238535669s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X...........XXXX...............X.X.....X.................] duration: 16.179614614s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:37:40 UTC) orbiter warning: context deadline exceeded Duration: 16.179639852s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.368514594s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:37:36 UTC) orbiter warning: context deadline exceeded Duration: 8.368540734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X...] duration: 3.801072482s trace: https://jaeger.dev.storj.io/trace/686bc369cc284c7 FAIL (2026-01-09 09:37:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80109677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.781890757s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:37:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.781918871s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...X...........XXXX...............X.X.....X................] duration: 7.255656125s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:37:14 UTC) orbiter warning: context deadline exceeded Duration: 7.255681695s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X..] duration: 5.001319248s trace: https://jaeger.dev.storj.io/trace/686bc369cc284c7 FAIL (2026-01-09 09:36:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001333576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.801317659s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.801347286s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 17.18715202s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:36:39 UTC) orbiter warning: context deadline exceeded Duration: 17.187174242s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX...X...........XXXX...............X.X.....X...............] duration: 10.241676973s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:36:22 UTC) orbiter warning: context deadline exceeded Duration: 10.241705468s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.856968784s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.856994382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X.] duration: 3.852850674s trace: https://jaeger.dev.storj.io/trace/686bc369cc284c7 FAIL (2026-01-09 09:36:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.852870302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 15.931373543s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:35:38 UTC) orbiter warning: context deadline exceeded Duration: 15.931388692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.95235668s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:35:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.952378893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX...X...........XXXX...............X.X.....X..............] duration: 6.832762142s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:35:04 UTC) orbiter warning: context deadline exceeded Duration: 6.832792871s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..X] duration: 5.001779899s trace: https://jaeger.dev.storj.io/trace/686bc369cc284c7 FAIL (2026-01-09 09:35:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001807151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.XX...X...........XXXX...............X.X.....X.............] duration: 7.737043055s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:34:45 UTC) orbiter warning: context deadline exceeded Duration: 7.737059226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.785871076s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:34:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.78589926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 18.052441528s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:34:27 UTC) orbiter warning: context deadline exceeded Duration: 18.052457559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX..] duration: 5.001002452s trace: https://jaeger.dev.storj.io/trace/478338a989bbda1 FAIL (2026-01-09 09:34:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001020515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XX...X...........XXXX...............X.X.....X............] duration: 20.244225761s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:34:17 UTC) orbiter warning: context deadline exceeded Duration: 20.244251079s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.631254565s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:34:09 UTC) orbiter warning: context deadline exceeded Duration: 15.631287026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX.] duration: 3.491400165s trace: https://jaeger.dev.storj.io/trace/478338a989bbda1 FAIL (2026-01-09 09:34:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491429121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 9.306215485s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:33:37 UTC) orbiter warning: context deadline exceeded Duration: 9.30624484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.867304987s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:33:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.867334683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXXX] duration: 5.000826522s trace: https://jaeger.dev.storj.io/trace/478338a989bbda1 FAIL (2026-01-09 09:33:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000847884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.875941971s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.875970605s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.XX...X...........XXXX...............X.X.....X...........] duration: 6.446386673s trace: https://jaeger.dev.storj.io/trace/402886061d167ef2 FAIL (2026-01-09 09:32:59 UTC) orbiter warning: context deadline exceeded Duration: 6.446414144s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XX.XXXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXXX] duration: 5.000751015s trace: https://jaeger.dev.storj.io/trace/2e42c60072c837d3 FAIL (2026-01-09 09:32:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000773748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.793649434s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.793677668s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 19.881663707s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:32:45 UTC) orbiter warning: context deadline exceeded Duration: 19.881691169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 21.7895433s trace: https://jaeger.dev.storj.io/trace/32dce7166c4df329 FAIL (2026-01-09 09:32:22 UTC) orbiter warning: context deadline exceeded Duration: 21.789565661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.880096542s trace: https://jaeger.dev.storj.io/trace/21422b5964c68bdd FAIL (2026-01-09 09:32:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 9 09:27:02.888560360 (833h58m2.759643415s) [upload=729472]: Starting upload Jan 9 09:27:02.888633259 (833h58m2.759716294s) [upload=729472]: Done starting upload Jan 9 09:27:02.888907641 (833h58m2.759990676s) [upload=729472]: Uploading segments... Jan 9 09:27:02.888968237 (833h58m2.760051292s) [upload=729472]: Got next segment. Inline: false Jan 9 09:27:02.889017601 (833h58m2.760100646s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 09:27:02.889022299 (833h58m2.760105344s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 09:27:03.135956776 (833h58m3.007039821s) [seg_pos={0 0}, upload=729472]: Begin upload segment... Jan 9 09:27:03.136013795 (833h58m3.00709684s) [seg_pos={0 0}, upload=729472]: Done begin upload segment. Jan 9 09:27:03.136044543 (833h58m3.007127588s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.136059551 (833h58m3.007142596s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137000804 (833h58m3.008083849s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137170998 (833h58m3.008254043s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.137752765 (833h58m3.00883581s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138204816 (833h58m3.009287861s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138256725 (833h58m3.00933978s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.138716510 (833h58m3.009799555s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139229998 (833h58m3.010313043s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.139683632 (833h58m3.010766677s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140179707 (833h58m3.011262752s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.140763929 (833h58m3.011846974s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141150876 (833h58m3.012233921s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.141626251 (833h58m3.012709287s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142208650 (833h58m3.013291695s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:03.142729391 (833h58m3.013812426s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Uploading piece... Jan 9 09:27:05.704344912 (833h58m5.575427957s) [upload=729472]: Next returned nil segment Jan 9 09:27:05.704360341 (833h58m5.575443386s) [upload=729472]: Waiting for error group managing segments... Jan 9 09:27:06.133601363 (833h58m6.004684408s) [noise=true , address=77.90.31.10:4517 , piece_id=5MGGWTHPNBMSYZ63YXLM2RDJKQNBFVBPGHYOY5MMLRBH27YPVYYQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.237788880 (833h58m6.108871925s) [noise=true , address=145.79.227.47:4266 , piece_id=Y3WOTXTJDD6IBEXKU2MXZSWFFQLZRAT4A7YTPQLM2XYVCND4UP3Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.238632687 (833h58m6.109715732s) [noise=true , address=145.79.227.44:4193 , piece_id=WU7X5ZX76IE7PCRS7P6OQC3ZAU4FMJQOEXXN2QNJPAAF4L6I6S7A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.243780239 (833h58m6.114863284s) [noise=true , address=145.79.227.5:4223 , piece_id=JFYTDEPUAKRICGX34GGHIDTQE6YD4DX3NOTWDAM2TIPZGFF2PDVA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.255218398 (833h58m6.126301433s) [noise=true , address=45.91.116.5:4299 , piece_id=UCB4FLLKL5YGZVQJQBAJXLPFWDG7N5JFEPZBABRA7637IXWJBIJA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.320674666 (833h58m6.191757711s) [noise=true , address=45.91.116.11:4118 , piece_id=QIG6D64M6LODYW3U56GKC3A5OHMCNVNBUOWXB4BUZEKQUA4ZJLIQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.321158788 (833h58m6.192241833s) [noise=true , address=145.79.227.49:4338 , piece_id=FVFPHST3ZQGWBFDTOT2VGUE4B624XZN47ZLAAEK7DHAG322ZSX2A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.322341570 (833h58m6.193424605s) [noise=true , address=145.79.227.28:4726 , piece_id=AQKPDPY7WCK2MQHW7VQA4GFNQQENUDNEVJIWE4TCPAJBYQUIR2UA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:06.994655956 (833h58m6.865739001s) [noise=true , address=77.90.31.21:4877 , piece_id=VUENTRDBFINI4276NHRILYX35DKE2HBMH5FXFMLZDEBPIOU2GP5Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.008139772 (833h58m6.879222817s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPETSZHKXDJBIVT5JPOVQ43NTF6TWL3CCPCYJZT6THLYFW7DTFBQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.011757821 (833h58m6.882840866s) [noise=true , address=145.79.227.3:4149 , piece_id=S3EVKK4NJQOVG4LWZ7FAPF5ID2HVRPXRXQP6JGD4CMF5OWO2POQA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:07.161320469 (833h58m7.032403514s) [noise=true , address=145.79.227.7:4293 , piece_id=KTVMVIK3KKTAPQ4IMQDJPGNWRBAUBJKPXU5WVW75QBE6WVOOYT7Q, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082847090 (833h58m7.953930135s) [noise=true , address=145.79.227.15:4873 , piece_id=BNXSHYO2CR36U55LUBWAX45WQHAPOGDTEG5G4KICHX7WXKRBB32A, seg_pos={0 0}, upload=729472]: Done uploading piece. err: <nil> Jan 9 09:27:08.082865885 (833h58m7.95394892s) [seg_pos={0 0}, upload=729472]: Segment reached optimal threshold of 13 pieces. Jan 9 09:27:08.083107996 (833h58m7.954191071s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083116703 (833h58m7.954199738s) [noise=true , address=145.79.227.41:4087 , piece_id=HTNWOT7U33VUFCNHY64GVRIEEER7MR6CAV7TZYPPZXYZHFGVKPNA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083278701 (833h58m7.954361737s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083296545 (833h58m7.95437958s) [noise=true , address=145.79.227.40:4049 , piece_id=JTALIBGAYIKZDUFYPL2D5HPDQPLLSK7GHT4JUUPXXS43BA2CP2YA, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083407267 (833h58m7.954490302s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Jan 9 09:27:08.083414450 (833h58m7.954497485s) [noise=true , address=145.79.227.48:4303 , piece_id=RQNZOLJVEYZRMH7BG4A6KRBU3VYBKHWCECVIYXJGEAJSQAE4YSPQ, seg_pos={0 0}, upload=729472]: Long tail cancellation due to optimal threshold Jan 9 09:27:08.083431512 (833h58m7.954514547s) [seg_pos={0 0}, upload=729472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 09:27:08.083498600 (833h58m7.954581645s) [upload=729472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 09:27:08.083504361 (833h58m7.954587396s) [upload=729472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 09:27:08.375829878 (833h58m8.246912913s) [upload=729472]: Done uploading segments... Jan 9 09:27:08.375849496 (833h58m8.246932541s) [upload=729472]: Upload finished. err: <nil> Duration: 5.880127381s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXX.XXXXXXXXX.XXXXXXXXXX.XXXX.XXXXX.XX...XXXXXX.X.XXXXXXXXXX] duration: 5.00149589s trace: https://jaeger.dev.storj.io/trace/4350c21c5c1e0182 FAIL (2026-01-09 09:32:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001516288s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)